# marginaleffects: Model to Meaning — Full Documentation
> How to interpret statistical models with `marginaleffects` for R and Python.
The `marginaleffects` package for R and Python helps analysts interpret statistical and machine learning models by computing predictions, comparisons (contrasts, marginal effects, treatment effects), slopes (partial derivatives), and hypothesis tests. The companion book "Model to Meaning" by Vincent Arel-Bundock (2026, CRC Press) is freely available at https://marginaleffects.com.
- Source: https://github.com/vincentarelbundock/marginaleffects
- Journal article: https://www.jstatsoft.org/article/view/v111i09
- Print book: https://routledge.com/9781032908724
- Table of contents: https://marginaleffects.com/llms.txt
---
# Book Chapters
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `predictions`, `comparisons`, `slopes`, `avg_predictions`, `avg_comparisons`, `avg_slopes` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `predictions`, `comparisons`, `slopes`, `avg_predictions`, `avg_comparisons`, `avg_slopes` for this chapter.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
# Models and meaning {#sec-goals}
- The best way to start a data analysis project is to set clear goals.
- This chapter explores four of the main objectives that data analysts pursue when they fit statistical models or deploy machine learning algorithms: model description, data description, causal inference, and out-of-sample prediction.
- To achieve these goals, it is crucial to articulate well-defined research questions, and to explicitly specify the statistical quantities---the estimands---that can shed light on those questions.
- Ideally, estimands should be expressed in the simplest form possible, on a scale that feels intuitive to stakeholders, colleagues, and domain experts.
- This chapter concludes by discussing some of the challenges that arise when trying to make sense of complex models.
- In many cases, the parameters of our models do not directly align with the estimands that actually interest us.
- Often, we must transform parameter estimates into quantities that directly inform our research questions, and that our audience will readily understand.
## Why fit a model? {#sec-challenge_goals}
- The first challenge that all researchers must take on is to transparently state what they hope to achieve with an analysis.
- The chapter surveys four goals an analyst can pursue: model description, data description, causal inference, and out-of-sample prediction.
- Each goal imposes different assumptions and requires different strategies for estimation and interpretation.
- Understanding which goal motivates an analysis helps determine the appropriate estimand and modeling approach.
- These four goals are not mutually exclusive, but conflating them can lead to misleading conclusions.
### Model description {#sec-challenge_model_description}
- The primary aim of model description is to understand how a fitted model behaves in different scenarios.
- The focus is on the internal workings of the model itself, rather than on making predictions or inferences about the sample or population.
- The analyst peeks inside the "black-box" to audit, debug, or test how the model reacts to different inputs.
- Model description aligns closely with the concepts of interpretability, explainability, and transparency in machine learning.
- It can provide some measure of reassurance that a fitted model works as intended and is suitable for deployment.
- To describe a model's behavior, the analyst might compute model-based predictions (expected values of the outcome for different subgroups).
- The analyst may also conduct counterfactual analyses to see how predictions change when predictor values are altered.
- For example, a financial analyst may compare what a model says about default risk for borrowers from different ethnic backgrounds, holding other predictors constant, to guard against algorithmic discrimination.
### Data description {#sec-challenge_data_description}
- Data description involves using statistical or machine learning models as tools to describe a sample, or to draw descriptive inference about a population.
- The objective is to explore and understand the characteristics of the data, often by summarizing their (potentially joint) distribution.
- Descriptive and exploratory data analysis can help analysts uncover new patterns, trends, and relationships.
- It can stimulate the development of theory or raise new research questions.
- Data description is arguably more demanding than model description, because it imposes additional assumptions.
- If the sample used to fit a model is not representative of the target population, or if the estimator is biased, descriptive inference may be misleading.
- To describe their data, an analyst might use a statistical model to compute the expected value of an outcome for different subgroups of the data.
### Causal inference {#sec-challenge_causality}
- In causal inference, the goal is to estimate the effect of an intervention (treatment, explanator, independent variable, or predictor) on some outcome (response or dependent variable).
- Causal inference is one of the most ambitious and challenging tasks in data analysis, typically requiring careful experimental design or statistical models that adjust for all confounders.
- The structural causal models approach (Judea Pearl) encodes causal relationships as equations, represented visually as directed acyclic graphs (DAGs), where nodes correspond to variables and arrows to causal effects.
- The potential outcomes framework (Neyman-Rubin Causal Model) defines causal effects as comparisons between what would happen to the same individual under different treatment conditions; the "fundamental problem of causal inference" is that we can only observe one potential outcome at a time.
- When conditions for causal identification are not met, some researchers replace causal language ("cause," "affect") with associational terms ("link," "correlation"), though others argue this introduces ambiguity about the researcher's actual goals.
- Transparency is recommended: candidly state interest in a causal effect while acknowledging limitations of the research design and data.
- The book presents estimands that can characterize either the "association" between two variables or the "effect" of one variable on another.
### Out-of-sample prediction
- Out-of-sample prediction and forecasting aim to predict future or unseen data points based on a model fit on existing data.
- This is particularly challenging due to the need to ensure that the model does not overfit the data and generalizes well.
- Out-of-sample prediction imposes additional requirements on the stability of the data distribution and the absence of changes in exogenous factors between the training sample and the target data.
- For example, a model trained to predict loan default probability may no longer perform well after an economic crisis changes personal finances across the population.
- Out-of-sample prediction is not the main focus of this book, but a case study on conformal prediction is presented later, covering a flexible strategy to make predictions and build intervals that cover a specified proportion of out-of-sample observations.
## What is your estimand? {#sec-goals_estimand}
- Once the overarching goal of an analysis is posed, the next step is to rigorously define the target of inquiry---the specific value that would shed light on the research question.
- An *estimand* is the quantity or parameter that we seek to learn.
- An *estimator* is the statistical method, algorithm, or mathematical formula applied to data to gain insight into the estimand.
- An *estimate* is the numerical result obtained by applying an estimator to data; it is our best guess of the estimand's true value based on available information.
- For example, if we want to know average adult height in a country (estimand), we might use the mean formula (estimator) to calculate 170cm from a random sample (estimate).
- The "Table 2 Fallacy" warns that two similar-looking statistical quantities, estimated by a single regression model, can have very different substantive interpretations; coefficients on control variables should not be individually interpreted as causal effects.
- The term "marginal effect" is ambiguous: in economics and political science it means a derivative (slope), while in other disciplines it means an average (integral) of unit-level estimates---two opposite mathematical operations.
- One of the main goals of this book is to help researchers overcome terminological ambiguity and define estimands clearly through a conceptual framework based on five simple questions.
## Making sense of parameter estimates {#sec-making_sense}
- Even if a fitted model is relatively simple, the parameter estimates it generates may not map directly onto an estimand that could inform the research question.
- For example, logistic regression coefficients are expressed as log odds ratios---the natural logarithm of a ratio-of-ratios-of-probabilities---which are very difficult to interpret substantively.
- Even straightforward probabilities are notoriously challenging to grasp intuitively; research in psychology and behavioral economics documents biases that distort how individuals perceive and make decisions based on them.
- The main contention of this book is that analysts should not focus on raw parameters but instead transform them into quantities that make more intuitive sense and shed light directly on the research question.
- Instead of reporting log odds ratios, analysts who fit logistic regressions should transform coefficients into predicted probabilities and compare predictions made with different predictor values.
- This transformation from logit coefficients to predicted probabilities is one example of a much more general, model-agnostic workflow applicable to over 100 different classes of statistical and machine learning models.
- By learning one conceptual framework and one set of tools, analysts can make sense of an extraordinarily large array of modeling contexts.
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `datagrid`, `predictions`, `comparisons`, `slopes`, `tt`, `source`, `options`, `subset` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `datagrid`, `predictions`, `comparisons`, `slopes`, `default_rng`, `DataFrame`, `normal`, `binomial` for this chapter.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
# Conceptual framework {#sec-framework}
- This chapter introduces a conceptual framework to aid the interpretation of a wide variety of statistical and machine learning models.
- Instead of focusing on fitted model parameters, analysts should convert those parameters into quantities that make more intuitive sense to readers and stakeholders.
- By applying *post-hoc* transformations, researchers can go from model to meaning.
- The workflow is both model-agnostic and consistent: every analysis starts from the same place by asking five critical questions.
- *Quantity:* Do we wish to estimate the level of a variable, the association between two (or more) variables, or the effect of a cause?
- *Predictors:* What predictor values are we interested in?
- *Aggregation:* Do we care about unit-level or aggregated estimates?
- *Uncertainty:* How do we quantify uncertainty about our estimates?
- *Test:* Which hypothesis or equivalence tests are relevant?
- These five questions lead to clear definitions of estimands and point to the specific software commands needed to run appropriate calculations.
## Quantity
- The parameters of a statistical model are often difficult to interpret, and they do not always shed direct light onto the research questions that interest us.
- In many contexts, it helps to transform parameter estimates into quantities with a more natural and domain-relevant meaning.
- For example, the analyst who fits a logistic regression model obtains coefficient estimates expressed as log odds ratios, a scale very difficult to reason about.
- Instead of struggling with complex amalgams of probabilities, analysts should transform estimates into more intuitive quantities, like predicted probabilities or risk differences.
- @sec-framework_quantity_theoretical_background exposes the theoretical underpinnings of *post hoc* transformations: the plug-in principle and the invariance property of maximum likelihood.
- Empirically-minded readers who are less interested in statistical theory may skip that part of the text.
- @sec-framework_three_quantities surveys the three classes of quantities of interest at the heart of this book: predictions, counterfactual comparisons, and slopes.
- These quantities are introduced briefly here, but given chapter-length treatments in Part II.
### Theoretical background {#sec-framework_quantity_theoretical_background}
- There are two primary theoretical justifications for post-estimation transformations: the plug-in principle and the invariance property of maximum likelihood estimators (MLE).
- The plug-in principle says that to infer some feature of a population, we can study the same feature in a sample, and plug in our sample estimate in lieu of the population value.
- Formally, if $\theta = \psi(F)$ is a statistical functional of the probability distribution $F$, and mild regularity conditions are satisfied, we can estimate $\theta$ using the sample analogue $\hat{\theta} = \psi(\hat{F}_n)$.
- As the number of observations increases, the empirical distribution function approximates the population distribution, and our estimate $\hat{\theta}$ tends to approach $\theta$.
- The plug-in principle justifies the workflow: fit a model, apply a function to coefficient estimates, and interpret the results as sample analogues to population characteristics.
- The invariance property of MLE states: if $\hat{\theta}$ is the MLE of $\theta$, then for any function $\psi(\theta)$, the MLE of $\psi(\theta)$ is $\psi(\hat{\theta})$.
- This means the desirable properties of MLEs -- consistency, efficiency, and asymptotic normality -- are preserved under transformation.
- In sum, post-estimation transformations are well-grounded in statistical theory, via the plug-in principle and the invariance property of MLE.
### Predictions, counterfactual comparisons, and slopes {#sec-framework_three_quantities}
- In this book, we target three broad classes of estimands: predictions, counterfactual comparisons, and slopes.
- Part II dedicates a full chapter to each of them, with many concrete examples drawing on real-world datasets.
- *A prediction* is the expected outcome of a fitted model for a given combination of predictor values.
- For example, using a linear model of children's heights as a function of age and caloric intake, we can predict that an 11-year-old who eats 1800 calories per day would have an expected height of 149.5 cm.
- *A counterfactual comparison* is a function of two predictions made with different predictor values; it measures the association between two variables, or the effect of one variable on another.
- Counterfactual comparisons can be expressed as differences, ratios, lift, odds ratios, or other functions of two predictions.
- When conditions for causal identification are satisfied, counterfactual comparisons can be interpreted as causal effects; otherwise they measure statistical association.
- *A slope* is the partial derivative of the regression equation with respect to a focal predictor; it measures the rate at which predictions change when a focal predictor changes by a small amount.
- In economics and political science, slopes are known as "marginal effects."
## Predictors {#sec-framework_grid}
- Predictions, counterfactual comparisons, and slopes are *conditional* quantities, which means that their values typically depend on all the predictors in a model.
- Whenever an analyst reports one of these statistics, they must imperatively disclose where it was evaluated in the predictor space.
- Answers to the question "for whom?" can be expressed in terms of profiles and grids.
- A *profile* is a specific combination of values for a focal predictor $X$ and a vector of control variables $\mathbf{Z}$; it is the set of predictor values for one observed or hypothetical individual.
- Profiles can be observed (actual data), synthetic (representative or interesting characteristics), or partially synthetic (an actual observation with a modified focal predictor).
- A *grid* is a collection of one or more profiles; defining the grid is a crucial step in model interpretation.
- The `datagrid()` function from the `marginaleffects` package constructs a variety of grids: empirical, interesting, representative, balanced, and counterfactual.
- The code examples use a simulated dataset with 10 observations on three variables: numeric (`Num`), binary (`Bin`), and categorical (`Cat`).
- R illustration code initializes simulated data using `library(marginaleffects)`, `library(tinytable)`, `set.seed(48103)`, `rnorm()`, `rbinom()`, and `sample()`.
- Python illustration code initializes the same dataset using `numpy` (`default_rng`, `normal`, `binomial`, `choice`) and `polars` (`DataFrame`), with `from marginaleffects import *`.
### Empirical grid {#sec-framework_grid_empirical}
- The empirical distribution is simply the observed dataset -- a grid composed of all actually observed profiles in the sample.
- Computing predictions on the empirical distribution is common practice, yielding one fitted value for each observation in the sample.
- When estimating counterfactual comparisons or slopes, analysts often start with the empirical grid, manipulate one focal predictor, and see how predicted outcomes are affected.
- Studying an empirical grid makes most sense when the observed sample is representative of the target population.
- When working with convenience samples with very different characteristics from the population, it may make sense to use a different grid type or apply weights as described in @sec-mrp.
- R illustration displays the simulated dataset using `dat |> tt()`.
- Python illustration displays the simulated dataset by calling `dat` directly in Polars.
### Interesting grid {#sec-framework_grid_interesting}
- If the analyst cares about units with specific profiles, they can use `datagrid()` to create customized grids of "interesting" predictor values.
- This is useful when one wants to compute a prediction or slope for an individual with given characteristics, such as a 50-year-old engineer from Belgium.
- By default, `datagrid()` fixes all variables to their means or modes, except for those variables that the analyst has explicitly defined.
- `datagrid()` also accepts functions to be applied to the variables in the original dataset (e.g., `range`, `mean`, `unique` in R).
- In both R and Python, specifying multiple values for multiple variables produces a grid with all combinations (the Cartesian product).
- R illustration creates an interesting grid with `datagrid(Bin = c(0, 1), newdata = dat)`, which fixes `Num` to its mean and `Cat` to its mode.
- R illustration also shows `datagrid(Num = range, Bin = mean, Cat = unique, newdata = dat)` to produce $2 \times 1 \times 3 = 6$ rows.
- Python illustration creates an interesting grid by passing lists and computed values: `datagrid(Bin=[0,1], newdata=dat)`.
- Python illustration also uses computed column values (e.g., `dat["Num"].max()`, `dat["Bin"].mean()`, `dat["Cat"].unique()`).
### Representative grid {#sec-framework_grid_representative}
- A representative grid is one where predictors are fixed to representative values, such as means, medians, or modes.
- This kind of grid is useful when the analyst wants to compute predictions, comparisons, or slopes for a typical or average individual.
- Representative grids allow us to compute quantities such as "fitted value at the median" or "marginal effect at the mean."
- Studying representative grids is useful for a measure of central tendency and for computational efficiency (one statistic for one profile).
- On the downside, nobody in the population may be exactly average on all dimensions, so the interpretation can be somewhat ambiguous.
- R illustration creates a representative grid with `datagrid(grid_type = "mean_or_mode", newdata = dat)`.
- Python illustration creates a representative grid with `datagrid(grid_type = "mean_or_mode", newdata = dat)`.
### Balanced grid {#sec-framework_grid_balanced}
- A balanced grid is built from all unique combinations of categorical variables, with all numeric variables held at their means.
- To create it, fix numeric variables at their means and create rows for each combination of categorical variable values (the Cartesian product).
- In the example, `Num` is held at its mean, and rows show all combinations of unique `Bin` and `Cat`.
- Balanced grids are often used to analyze the results of factorial experiments in convenience samples, where the empirical distribution is not representative of the target population.
- Balanced grids are used by default in the `emmeans` post-estimation software, and we will see them in action in @sec-predictions_aggregation when computing marginal means.
- R illustration creates a balanced grid with `datagrid(grid_type = "balanced", newdata = dat)`.
- Python illustration creates a balanced grid with `datagrid(grid_type = "balanced", newdata = dat)`.
### Counterfactual grid {#sec-framework_grid_counterfactual}
- The last type of grid to consider is counterfactual: the entire dataset is duplicated, creating one copy for every combination of values that the analyst supplies.
- In the example, `Bin` must take values 0 or 1, so two copies of the full dataset are created (one with `Bin=0`, one with `Bin=1`), for a total of 20 rows.
- All other variables are held at their observed values.
- Each original row has an exact duplicate that differs only in the counterfactual `Bin` variable; the `rowidcf` column holds row indices.
- This kind of duplication is essential in chapters on counterfactual comparisons and G-computation, where we explore counterfactual analysis and causal inference.
- R illustration creates a counterfactual grid with `datagrid(Bin = c(0, 1), grid_type = "counterfactual", newdata = dat)`.
- R illustration inspects the first three rows of each counterfactual version with `subset(g, rowidcf %in% 1:3) |> tt()`.
- Python illustration creates a counterfactual grid with `datagrid(Bin=[0,1], grid_type = "counterfactual", newdata = dat)`.
- Python illustration inspects rows by filtering on the `rowidcf` column with `g.filter(pl.col("rowidcf") < 3)`.
## Aggregation {#sec-framework_aggregation}
- Predictions, counterfactual comparisons, and slopes are conditional quantities that depend on the values of all predictors; computing them over a grid yields one estimate per row.
- If a grid has many rows, the large number of estimates can be unwieldy, so analysts may aggregate unit-level estimates into macro-level summaries.
- No aggregation: Unit-level estimates.
- Overall average: Average of unit-level estimates.
- Subgroup averages: Average of unit-level estimates within subgroups of the data.
- Weighted averages: Weighted average of unit-level estimates, where weights could be sampling weights or the inverse probability of treatment assignment.
- Aggregated estimates are common: average predictions, marginal means, average counterfactual comparisons (ATE, ATT, ATU), and average slopes (average marginal effects).
- Aggregated estimates tend to be easier to interpret and estimated with greater precision, but they can mask interesting variation across the sample (e.g., positive and negative effects cancelling out).
## Uncertainty
- Whenever we report quantities of interest derived from a statistical model, it is essential to provide estimates of our uncertainty.
- Without standard errors or confidence intervals, readers cannot assess if reported values are genuine or the product of chance.
- The `marginaleffects` package offers four primary methods for quantifying uncertainty: the delta method (default), bootstrap, simulation-based inference, and conformal prediction.
- The delta method approximates the variance of a function of random variables; it is fast, flexible, and can be paired with robust variance estimates, but relies on a linear approximation and asymptotic normality.
- The bootstrap generates empirical distributions by repeatedly resampling from the observed data; it is useful when the delta method's assumptions are not met.
- Simulation-based inference draws simulated coefficients from an assumed distribution and computes quantities of interest repeatedly; it is effective for complex models and provides intuitive visualizations of uncertainty.
- Conformal prediction provides valid *prediction* intervals (rather than *confidence* intervals) under minimal assumptions, using split-sample strategies.
- @sec-uncertainty is entirely dedicated to uncertainty quantification with intuition, technical details, and hands-on demonstrations.
## Test {#sec-framework_test}
- Once we have computed a quantity of interest and its standard error, we can conduct a test to check if a hypothesis or conjecture is correct.
- Null hypothesis tests determine if there is sufficient evidence to *reject* a presumed statement about a quantity of interest.
- Examples of null hypothesis tests: the difference between two regression coefficients equals one; the predicted number of goals in a game is two; the effect of a medication on blood pressure is null; the association between education and income is zero.
- Equivalence tests provide evidence that the difference between an estimate and some reference value is "negligible" or "unimportant."
- Examples of equivalence tests: one regression coefficient is practically equivalent to another; the predicted probability is not meaningfully different from a target; two treatment effects are essentially the same.
- Null hypothesis tests establish a difference; equivalence tests establish a similarity or practical equivalence.
- The `marginaleffects` package computes null hypothesis and equivalence tests on raw parameter estimates, (non-)linear combinations of parameters, and on all estimated quantities: predictions, counterfactual comparisons, and slopes.
- @sec-hypothesis is entirely dedicated to hypothesis and equivalence testing.
## Summary
- *Quantity:* Do we wish to estimate the level of a variable, the association between two (or more) variables, or the effect of a cause?
- *Predictors:* What predictor values are we interested in?
- *Aggregation:* Do we care about unit-level or aggregated estimates?
- *Uncertainty:* How do we quantify uncertainty about our estimates?
- *Test:* Which hypothesis or equivalence tests are relevant?
- This chapter was written to address two problems: (a) the parameters of a statistical model are often difficult to interpret, and (b) analysts often fail to rigorously define the statistical quantities (estimands) and tests that can shed light on their research questions.
- We solve these problems by transforming parameter estimates into quantities with a straightforward interpretation and a direct link to research goals.
- The analysis workflow implied by these five questions is extremely flexible and can be operationalized via the consistent user interface of the `marginaleffects` package for R and Python.
# Examples
- The examples below are drawn from the original chapter and illustrate representative workflows.
## Example 1: Creating an interesting grid
```r
library(marginaleffects)
library(tinytable)
set.seed(48103)
N <- 10
dat <- data.frame(
Num = rnorm(N),
Bin = rbinom(N, size = 1, prob = 0.5),
Cat = sample(c("A", "B", "C"), size = N, replace = TRUE)
)
# Create an interesting grid with Bin fixed to 0 and 1,
# other variables at their means/modes
datagrid(Bin = c(0, 1), newdata = dat) |> tt()
```
## Example 2: Creating a counterfactual grid
```r
# Duplicate the full dataset for each value of Bin
g <- datagrid(
Bin = c(0, 1),
grid_type = "counterfactual",
newdata = dat
)
nrow(g) # 20 rows: 10 original x 2 counterfactual values
# Inspect first three rows of each counterfactual copy
subset(g, rowidcf %in% 1:3) |> tt()
```
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `hypotheses`, `coef`, `summary`, `vcov`, `pnorm`, `aggregate`, `get_dataset`, `lm` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `hypotheses`, `ols`, `fit`, `to_pandas`, `group_by`, `agg`, `col`, `mean` for this chapter.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
## Summary
This chapter introduced two classes of statistical testing procedures: null hypothesis and equivalence tests.
A null hypothesis test allows us to determine if there is enough evidence to reject the hypothesis that a parameter (or function of parameters) is *equal* to a given value.
Examples of statements that could be rejected by a null hypothesis test include:
* The predicted wages of college and high school graduates are equal.
* The effect of a new drug on a health outcome is zero.
* A marketing campaign has the same effect on sales in rural or urban areas.
When a null hypothesis test indicates that we can reject statements like these (small $p$ value), *we establish a difference.*
An equivalence test allows us to determine if there is enough evidence to reject the hypothesis that a parameter (or function of parameters) is *meaningfully different* from a benchmark.
Examples of statements that could be rejected by an equivalence test include:
* The difference in wages between college and high school graduates is considerable.
* The effect of a new drug on a health outcome is meaningfully different from the effect of an existing treatment.
* The effect of a marketing campaign on consumption is much larger than zero.
When an equivalence test indicates that we can reject statements like these (small $p$ value), *we establish a similarity.*
All the main `marginaleffects` functions include `hypothesis` and `equivalence` arguments. These arguments make it easy to conduct null hypothesis and equivalence tests on any of the quantities estimated by the package---predictions, counterfactual comparisons, and slopes---as well as on arbitrary functions of those quantities.
# Hypothesis and equivalence tests {#sec-hypothesis}
- This chapter introduces two complementary statistical testing procedures: null hypothesis tests and equivalence tests.
- A null hypothesis test assesses whether we can reject the possibility that a population parameter (or function of parameters) takes on a specific value, such as zero.
- An equivalence test flips the logic: instead of establishing a difference, it makes a case for similarity by testing whether an estimate falls within a pre-defined interval of practical equivalence.
- Null hypothesis tests answer questions like: Does cognitive-behavioral therapy have a non-zero effect on depression? Is the effect of a new drug different from the effect of an existing treatment?
- Equivalence tests answer questions like: Is the effect of a generic drug equivalent to that of the branded version? Is the effect of a marketing campaign on consumption negligible?
- Both types of tests can be applied to model parameters, predictions, counterfactual comparisons, and slopes -- any quantity of interest studied in this book.
- An important distinction is between statistical significance (unlikely to occur by chance if the null hypothesis is true) and practical significance (important real-world implications). Many results are statistically significant without having much practical significance.
- The main dataset used for illustration comes from Thornton (2008), a randomized controlled trial studying whether small monetary incentives encourage people to seek information about their HIV status.
- The outcome variable is binary (`outcome` = 1 if the participant traveled to learn their test result), and predictors include `incentive` (treatment assignment), `distance` from the test center, `village`, and `agecat` (age in three categories: <18, 18 to 35, >35).
- A linear probability model is estimated with `agecat` levels as predictors and no intercept, so that the coefficients equal subgroup means of the outcome variable.
## Null hypothesis {#sec-hypothesis_null_hypothesis}
- The null hypothesis test is a statistical method to determine if there is sufficient evidence to reject a presumed statement about a population parameter.
- The null hypothesis $H_0$ represents a default claim, usually suggesting no effect or no difference. For example, $H_0$ might state that the mean of a population equals a specific value.
- To conduct a null hypothesis test: (1) choose a null hypothesis $H_0$, (2) pick a test statistic with a known sampling distribution (e.g., $Z$ or $t$), (3) compute the test statistic from observed data, and (4) compare it to the assumed distribution under $H_0$.
- The standard Wald approach constructs a $Z$ statistic: $Z = \frac{h(\hat{\theta}) - H_0}{\sqrt{\hat{V}[h(\hat{\theta})]}}$, where $h(\hat{\theta})$ is a quantity of interest and $\hat{V}$ is its estimated variance.
- When $|Z|$ is large, the numerator (distance between estimate and null) is large relative to the denominator (uncertainty in the estimate), so we can reject $H_0$.
- By default, R and Python `summary()` functions report null hypothesis tests against $H_0$: coefficient = 0. This default may not always be substantively meaningful.
- For the Thornton data, testing whether the proportion of minors seeking HIV results equals zero is not interesting -- a more meaningful null hypothesis should be chosen.
- The `marginaleffects` package makes it easy to construct alternative test statistics using the `hypotheses()` function.
### Choice of null hypothesis
- Instead of testing against zero, analysts should specify a null hypothesis that represents a meaningful benchmark for their research question.
- The `hypotheses()` function with its `hypothesis` argument allows testing all model coefficients against any numeric null value (e.g., `hypothesis = 0.5` to test whether each coefficient differs from 0.5).
- In the running example, testing $H_0$: coefficient = 0.5 asks whether the probability of retrieving one's HIV test result differs from a coin flip.
- When the $Z$ statistics are large in absolute terms, we can reject the null hypotheses. If the true probability were 50/50, we would be very unlikely to observe data like these.
- Wald-style $p$ values can also be computed manually: the two-tailed $p$ value measures the area under the tails of the normal distribution beyond $|Z|$.
- In R, this is done with `pnorm(-abs(z)) * 2`; in Python, with `norm.cdf(-np.abs(z)) * 2`.
### Linear and non-linear hypothesis tests
- Analysts often wish to compare different quantities to one another rather than testing against a simple numeric null like 0 or 0.5.
- The `hypothesis` argument accepts equation-style strings where terms start with `b` followed by the position of the estimate (e.g., `"b3 - b1 = 0"` in R, `"b2 - b0 = 0"` in Python due to 0-based indexing).
- For example, testing $H_0: \beta_1 = \beta_3$ checks whether the probability of seeking HIV results is the same in the <18 and >35 age groups.
- Whether to reject the null depends on the chosen significance threshold. If mistakenly rejecting the null has costly consequences, a more stringent threshold should be used.
- Ratio tests are also supported: `"b3 / b1 = 1"` tests whether the ratio of two coefficients equals one, rather than testing whether their difference equals zero.
- The `hypothesis` argument supports arbitrary (potentially non-linear) equations with multiple estimates, such as `"b2^2 * exp(b1) = 0"` or `"b1 - (b2 * b3) = 2"`.
- A formula-based shortcut interface is available for common tests: `hypothesis = difference ~ reference` computes differences between every coefficient and the first estimate.
- Other shortcuts include `ratio ~ sequential` to compute the ratio of each coefficient to the one that immediately precedes it.
### Multiple comparisons and joint hypothesis tests
- When multiple hypothesis tests are performed simultaneously, the likelihood of at least one Type I error (falsely rejecting a true null) increases. This is the multiple comparisons problem.
- Statisticians have proposed correction procedures including Bonferroni, Holm, and Westfall corrections to adjust for multiple comparisons.
- The `hypotheses()` function can apply these strategies via the `multcomp` argument (e.g., `multcomp = "holm"`), reporting corrected $p$ values and family-wise confidence intervals.
- The `hypotheses()` function also supports joint hypothesis tests via the `joint` and `joint_test` arguments, testing whether several quantities of interest are jointly/simultaneously equal to zero.
- Documentation and examples for joint hypothesis tests are available on the marginaleffects.com website.
## Equivalence {#sec-hypothesis_equivalence}
- An equivalence test determines if an estimate is "practically equivalent" to a benchmark, within a specified margin of equivalence.
- Unlike traditional null hypothesis tests (which reject a point null), equivalence tests reject the null hypothesis that the estimand lies *outside* an interval of practical equivalence.
- The Two One-Sided Test (TOST) is the standard equivalence testing procedure, conducted in six steps: (1) define and estimate a quantity of interest $\theta$; (2) choose a significance threshold $\alpha$; (3) define an interval of equivalence $[a, b]$ using subject-matter knowledge; (4) conduct a non-inferiority test ($H_0: \theta < a$); (5) conduct a non-superiority test ($H_0: \theta > b$); (6) check if the maximum of both $p$ values is below $\alpha$.
- The definition of the equivalence interval is a substantive question, not a statistical one. It depends on the field, research question, costs, and domain expertise.
- In the running example, the equivalence interval is $[-0.05, 0.05]$: if the difference in probability of seeking HIV results between age groups falls within this range, it is considered practically equivalent to zero.
- To conduct a TOST, add the `equivalence` argument to `hypotheses()` (e.g., `equivalence = c(-0.05, 0.05)` in R or `equivalence=[-0.05, 0.05]` in Python).
- In the example, non-inferiority is established (we can reject $\theta < -0.05$) but non-superiority is not (we cannot reject $\theta > 0.05$), so we cannot conclude equivalence overall.
- The same TOST procedure can be applied to predictions, counterfactual comparisons, and slopes -- not just differences between coefficients. The equivalence interval can be centered anywhere, not just around zero.
# Examples
- The examples below are drawn from the original chapter and illustrate representative workflows.
## Example 1: Testing coefficients against a custom null hypothesis
```r
library(marginaleffects)
dat <- get_dataset("thornton")
mod <- lm(outcome ~ agecat - 1, data = dat)
# Test all coefficients against H0: coefficient = 0.5
hypotheses(mod, hypothesis = 0.5)
```
```python
from marginaleffects import *
from statsmodels.formula.api import ols
dat = get_dataset("thornton")
mod = ols("outcome ~ agecat - 1", data=dat.to_pandas()).fit()
hypotheses(mod, hypothesis=0.5)
```
## Example 2: Linear hypothesis test comparing two coefficients
```r
# Test H0: beta_3 - beta_1 = 0
hypotheses(mod, hypothesis = "b3 - b1 = 0")
```
```python
# Test H0: b2 - b0 = 0 (0-based indexing)
hypotheses(mod, hypothesis = "b2 - b0 = 0")
```
## Example 3: Equivalence test using TOST
```r
# Test whether the difference between coefficients for
# the 18-35 and >35 age groups falls within [-0.05, 0.05]
hypotheses(mod,
hypothesis = "b3 - b2 = 0",
equivalence = c(-0.05, 0.05))
```
```python
hypotheses(mod,
hypothesis="b2 - b1 = 0",
equivalence=[-0.05, 0.05])
```
---
## Start Here: Manual Pages
- R: open function help with `?predictions`, `?avg_predictions`, `?plot_predictions`, `?datagrid`; also see `?inferences` for resampling-based uncertainty.
- Python: inspect docstrings with `help(predictions)`, `help(avg_predictions)`, `help(plot_predictions)`, `help(datagrid)` from the `marginaleffects` package.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
## Original Chapter Summary
This chapter defined a "prediction" as the outcome expected by a fitted model for a given combination of predictor values. A prediction is a useful descriptive quantity. It is an expectation or best guess for different individuals, units, or subgroups of the population of interest.
The `predictions()` function from the `marginaleffects` package computes predictions for a wide range of models. `avg_predictions()` aggregates unit-level predictions. `plot_predictions()` displays predictions visually.
To clearly define predictions and attendant tests, analysts must make five decisions.
First, the *Quantity*.
- Predictions can be computed on different scales, depending on the type of statistical model. In generalized linear models (GLM), for example, one can make predictions on the "link" or "response" scales. In most cases, analysts should report predictions on the same scale as the outcome variable, since this is most natural for readers.
- The scale of predictions is controlled by the `type` argument.
Second, the *Predictors*.
- Predictions are conditional quantities, that is, they depend on the values of all the predictors in a model.
- Analysts can make predictions for different combinations of predictor values, or grids: empirical, interesting, representative, balanced, or counterfactual.
- The predictor grid is defined by the `newdata` argument and the `datagrid()` function.
Third, the *Aggregation*.
- To simplify the presentation of results, it often makes sense to report average predictions. Analysts can choose between different aggregation schemes:
- Unit-level predictions (no aggregation)
- Average predictions
- Average predictions by subgroup
- Weighted average of predictions
- Predictions can be aggregated using the `avg_predictions()` function and the `by` argument.
Fourth, the *Uncertainty*.
- In `marginaleffects`, the `vcov` argument allows analysts to report classical, robust, or clustered standard errors around predictions.
- The `inferences()` function can compute uncertainty intervals via bootstrapping, simulation-based inference, or conformal prediction.
Fifth, the *Test*.
- A null hypothesis test aims to determine if a prediction (or a function of predictions) is different from a null hypothesis value. For example, an analyst may wish to check if two predictions are different from one another. Null hypothesis tests can be conducted using the `hypothesis` argument.
- An equivalence test aims to determine if a prediction (or a function of predictions) is similar to a reference value. Equivalence tests can be conducted using the `equivalence` argument.
# Predictions {#sec-predictions}
- A prediction is the outcome expected by a fitted model for a given combination of predictor values. This is akin to a "fitted value," but not a "forecast" or "out-of-sample prediction."
- Model-based predictions answer questions like: What is the expected probability that a 50-year-old smoker develops heart disease? What is the expected turnout in municipal elections?
- Predictions are an intrinsically interesting descriptive quantity and a fundamental building block for comparisons and slopes.
- Predictions can be computed on different scales (e.g., "link" or "response" in GLMs). Analysts should generally report on the response scale.
- The scale of predictions is controlled by the `type` argument.
- Predictions are conditional quantities: they depend on the values of all predictors in a model.
- Analysts can make predictions for different grids of predictor values: empirical, interesting, representative, balanced, or counterfactual.
- The chapter proceeds through the five components of the conceptual framework: quantity, predictors, aggregation, uncertainty, and tests.
## Quantity {#sec-predictions_quantity}
- The chapter illustrates predictions using a logistic regression model on the Thornton (2008) HIV dataset.
- The model predicts the probability that a participant travels to a test center to learn their HIV status, as a function of monetary incentive and age category.
- The logistic function $g(x) = \frac{1}{1 + e^{-x}}$ maps the linear predictor to the $[0,1]$ probability scale.
- Link scale predictions (log odds) can be unintuitive; response scale predictions (probabilities) are easier to interpret and compare to observed outcomes.
- The `predictions()` function from `marginaleffects` automates prediction across 100+ model classes.
- To use `predictions()`, build a data frame (grid) of predictor values and pass it via the `newdata` argument.
- The `type` argument controls whether predictions are on the link or response scale; the default is the response scale.
- Manual computation of predictions is useful pedagogically but labor-intensive; `predictions()` is the recommended approach.
## Predictors {#sec-predictions_predictors}
- Predictions are conditional quantities: to compute a prediction, the analyst must fix all right-hand-side variables (choose a grid).
- The choice of grid depends on the researcher's goals; profiles may correspond to observed data or hypothetical units.
- Five grid types are discussed: empirical, interesting, representative, balanced, and counterfactual.
- The `newdata` argument and `datagrid()` function are the primary tools for defining grids.
- Different grids can produce substantially different results, so the choice of grid matters.
### Empirical grid
- By default, `predictions()` uses the full original dataset as a grid (the empirical distribution of predictors), computing fitted values for every row.
- The returned object is a standard data frame with fitted values, confidence intervals, and test statistics.
- By default, $p$ values in the output correspond to a test against a null of zero.
- Standard data manipulation tools (subsetting, column extraction) can be applied to the output.
- This is equivalent to computing classical "fitted values."
### Interesting grid
- Analysts can build a custom grid of predictor values with particular scientific or commercial interest.
- The `datagrid()` function creates a grid with unspecified variables held at their means (numeric) or modes (categorical).
- Functions can be passed to `datagrid()` to programmatically define variable values (e.g., `unique`, `max`).
- When `datagrid()` is called inside a `marginaleffects` function, the `model` argument can be omitted.
- The grid is passed to `predictions()` via the `newdata` argument.
### Representative grid
- A "Prediction at the Mean" computes a prediction for a hypothetical individual with mean numeric and modal categorical characteristics.
- This is achieved via `datagrid()` with no specified variables, or the `newdata = "mean"` shortcut.
- Representative grids can be useful, but the "average" individual may be fictional and the prediction may not be practically relevant.
- Analysts should consider whether anyone in the sample actually resembles the representative profile.
- This approach returns a single prediction for one representative unit.
### Balanced grid
- A balanced grid includes one row for each combination of unique values of categorical/binary predictors, with numeric variables at their means.
- Created via `datagrid()` with `unique` for categoricals and `mean` for numerics, or the `newdata = "balanced"` shortcut.
- Commonly used in experimental settings to give equal weight to each treatment combination.
- Balanced grids are important for computing marginal means.
- Results from a balanced grid can differ substantially from those on an empirical grid.
### Counterfactual grid {#sec-predictions_grid_counterfactual}
- A counterfactual grid duplicates the full dataset, fixing a focal variable to each of its possible values.
- This allows answering: "What would the predicted outcomes be if everyone received treatment vs. control?"
- Created via `datagrid()` with `grid_type="counterfactual"`, or via the `variables` argument of `predictions()`.
- Each individual gets a prediction under each counterfactual scenario, enabling individual-level causal reasoning.
- In the Thornton example, every participant's predicted probability increases when `incentive` is set to 1 vs. 0.
## Aggregation {#sec-predictions_aggregation}
- Unit-level predictions for large datasets can be unwieldy; aggregation simplifies results.
- An "average prediction" is a two-step process: compute fitted values for each row, then average them.
- `avg_predictions()` is a convenience wrapper that computes average predictions directly.
- The `by` argument allows averaging within subgroups (e.g., by age category or treatment status).
- "Marginal means" average predictions across a balanced grid, giving equal weight to each treatment combination; this is common in experimental analysis and is the default in packages like `emmeans`.
- Results differ between empirical and balanced grids because unequal group sizes change the weighting.
- Counterfactual aggregation uses the `variables` argument: duplicate the data under each treatment value and average predictions within each group.
- The choice of aggregation grid (empirical, balanced, counterfactual) can substantially affect results.
## Uncertainty {#sec-predictions_uncertainty}
- The `conf_level` argument controls the confidence interval width (default: 95%).
- The `vcov` argument specifies the type of standard errors: classical (default), heteroskedasticity-consistent (e.g., `"HC3"`), or clustered (e.g., `vcov = ~village`).
- Classical standard errors assume errors are independently and identically distributed.
- Robust standard errors (e.g., HC3) relax the homoskedasticity assumption.
- Clustered standard errors account for within-group correlation in the data.
- The `inferences()` function supports bootstrap, simulation-based inference, and conformal inference.
- Different uncertainty methods yield slightly different intervals but generally remain in the same ballpark.
## Test {#sec-predictions_test}
- This section covers formal statistical comparison of predictions across subgroups.
- The `hypothesis` argument is used for null hypothesis tests on predictions or functions of predictions.
- String syntax like `"b3 - b2 = 0"` tests whether the difference between specific group predictions equals zero.
- Formula syntax like `difference ~ sequential` compares each group to the preceding one; `difference ~ reference` compares each to the baseline group.
- A vertical bar in the formula (e.g., `difference ~ sequential | incentive`) performs comparisons within subgroups.
### Null hypothesis tests
- Compare average predictions across subgroups using the `hypothesis` argument.
- String-based hypotheses (e.g., `"b3 - b2 = 0"`) allow testing specific contrasts.
- Formula-based hypotheses (e.g., `difference ~ sequential`) automate pairwise or sequential comparisons.
- The `|` operator enables within-subgroup comparisons (e.g., sequential differences within each `incentive` group).
- Multiple comparisons can be conducted simultaneously.
### Equivalence tests
- An equivalence test determines if a difference is small enough to be considered negligible.
- The `equivalence` argument specifies an interval of practical equivalence (e.g., `c(-0.1, 0.1)`).
- A small $p$ value in an equivalence test means we can reject the null that the difference is large or meaningful.
- This flips the logic of a standard null hypothesis test: instead of testing "is this different from zero?" we test "is this close enough to zero?"
- The interval of practical equivalence should be chosen based on domain-specific reasoning.
## Visualization {#sec-predictions_visualization}
- The `plot_predictions()` function visualizes predictions with a syntax paralleling other `marginaleffects` functions.
- Three main strategies: unit-level distributions, marginal (averaged) predictions, and conditional predictions.
- Output is a `ggplot2` object in R and a `plotnine` object in Python, making customization straightforward.
- The `draw = FALSE` argument returns the raw data frame used for plotting, for fully custom visualizations.
### Unit predictions
- Histograms and ECDF plots display the full distribution of individual-level predictions.
- The `predictions()` output is a standard data frame, compatible with any plotting library.
- Displaying full distributions (rather than summaries) helps convey heterogeneity across predictor values.
- Frank Harrell argues analysts should avoid one-number summaries and instead show the full distribution.
- The `patchwork` package in R (or similar tools) can combine multiple plots.
### Marginal predictions
- The `by` argument in `plot_predictions()` computes and displays average predictions by subgroup.
- This is equivalent to plotting the results of `avg_predictions()` with the `by` argument.
- The `newdata` argument can be used to plot marginal means from a balanced grid.
- Marginal prediction plots clearly show group-level differences (e.g., treatment vs. control).
- Multiple `by` variables can be specified to display interaction patterns.
### Conditional predictions
- The `condition` argument builds a grid of representative values and plots predictions without averaging.
- Numeric variables are spread over an equally spaced grid from min to max; categorical variables use all unique values.
- Other predictors not in `condition` are held at their means or modes.
- Multiple conditioning variables can be passed as a vector or list to create multi-panel or multi-line plots.
- Specific values can be fixed via a named list (e.g., `condition = list("distance", "agecat" = ">35")`).
### Customization
- In R, `plot_predictions()` returns a `ggplot2` object; in Python, it returns a `plotnine` object.
- Standard `ggplot2`/`plotnine` functions (themes, scales, labels, limits) can be applied to customize appearance.
- The `rug = TRUE` argument adds a rug plot showing the distribution of predictor values.
- Setting `draw = FALSE` returns a data frame, enabling fully custom plot construction with any graphics library.
- This approach separates data generation from visualization, giving the analyst maximum flexibility.
# Examples
## Example 1: Predictions on a custom grid
```r
library(marginaleffects)
dat <- get_dataset("thornton")
mod <- glm(outcome ~ incentive + agecat + distance,
data = dat, family = binomial)
predictions(mod,
newdata = datagrid(agecat = "18 to 35", incentive = c(0, 1)))
```
```python
from marginaleffects import *
from statsmodels.formula.api import logit
dat = get_dataset("thornton").drop_nulls(subset=["incentive"])
mod = logit("outcome ~ incentive + agecat + distance",
data=dat.to_pandas()).fit()
predictions(mod,
newdata=datagrid(agecat="18 to 35", incentive=[0, 1]))
```
## Example 2: Average predictions by subgroup
```r
avg_predictions(mod, by = "agecat")
```
```python
avg_predictions(mod, by="agecat")
```
## Example 3: Plotting conditional predictions
```r
plot_predictions(mod,
condition = c("distance", "incentive", "agecat"))
```
```python
plot_predictions(mod,
condition={"distance": None, "incentive": None, "agecat": None}).show()
```
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `comparisons`, `avg_comparisons`, `plot_comparisons`, `datagrid`, `inferences`, `predictions`, `avg_predictions` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `comparisons`, `avg_comparisons`, `plot_comparisons`, `datagrid`, `avg_predictions`, `get_dataset` for this chapter.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
## Summary
This chapter defined a "counterfactual comparison" as a function of two or more model-based predictions made with different predictor values. These comparisons measure the strength of association between variables or, if appropriate identification assumptions are met, they quantify causal effects.
The `comparisons()` function from the `marginaleffects` package computes counterfactual comparisons for a wide range of models, and `avg_comparisons()` aggregates them across units or groups. Analysts can visualize these comparisons with the `plot_comparisons()` function or standard visualization tools like `ggplot2`.
To define, compute, and interpret counterfactual comparisons, analysts must make five decisions:
- **Quantity**: Counterfactual comparisons are defined along two dimensions. What change in focal predictor are we interested in (e.g., 0 to 1, increase of 1 unit or 1 standard deviation, change between specific values)? This is specified by the `variables` argument. What function do we use to compare counterfactual predictions (difference, ratio, lift, etc.)? This is specified by the `comparison` argument.
- **Predictors**: Counterfactual comparisons are conditional quantities that depend on the values of all predictors in a model. Analysts can evaluate comparisons for different predictor grids, including empirical, balanced, interesting, or counterfactual grids, defined by the `newdata` argument and the `datagrid()` function.
- **Aggregation**: Analysts can report unit-level comparisons (specific to each observation) or aggregated comparisons (average effects across the population or subgroups, computed with `avg_comparisons()` and the `by` argument).
- **Uncertainty**: The uncertainty around counterfactual comparisons can be estimated using classical or robust standard errors, bootstrap, or simulation-based inference. Standard errors and confidence intervals are handled by the `vcov` and `conf_level` arguments, or via the `inferences()` function.
- **Test**: Hypothesis tests can check if counterfactual comparisons differ significantly from a null value or from one another, via the `hypothesis` argument. Equivalence tests can establish whether comparisons are practically equivalent using the `equivalence` argument.
# Counterfactual comparisons {#sec-comparisons}
- Many important research questions can be expressed as comparisons between hypothetical worlds: would outcomes differ if a treatment were applied or withheld?
- A counterfactual comparison is a function of two or more model-based predictions, made with different predictor values.
- When conditions for causal identification are satisfied, a counterfactual comparison can be interpreted as a causal effect of $X$ on $Y$.
- When causal conditions are not met, counterfactual comparisons remain useful as descriptive measures of the strength of association between variables, holding other variables constant.
- Counterfactual comparisons can be expressed on the scale of the outcome variable, avoiding complex functions like log-odds ratios.
- Counterfactual comparisons map directly onto what many people mean by "the effect of a treatment": what change do we expect in the outcome when a predictor changes?
- The `marginaleffects` package makes it easy to compute counterfactual comparisons in a model-agnostic fashion, applying similar post-estimation steps regardless of the model type.
- The chapter uses a logistic regression model fitted to @Tho2008 HIV data as a running example: the outcome is whether a participant sought their HIV status, and the treatment is a monetary incentive, interacted with age category and distance from the test center.
## Quantity {#sec-comparisons_quantity}
- A counterfactual comparison is a function of two or more model-based predictions, made with different predictor values.
- To operationalize this, analysts must make three decisions: (1) what is the focal predictor, (2) how does the focal predictor differ between counterfactual worlds, and (3) what function compares the predicted outcomes.
- When the variable $X$ is set to value $x$, the model-based prediction is $\hat{Y}_{X=x}$.
- The predicted outcome can change when $X$ increases by 1 unit, by one standard deviation, from minimum to maximum, or between two specific values.
- A simple difference between predictions is often the best starting point because it is easy to grasp intuitively.
- Alternatives to differences include the ratio ($\hat{Y}_{X=b}/\hat{Y}_{X=a}$) and lift ($(\hat{Y}_{X=b} - \hat{Y}_{X=a})/\hat{Y}_{X=a}$).
- When the predicted outcome is a probability, $\hat{Y}_{X=b}-\hat{Y}_{X=a}$ is called a risk difference, $\hat{Y}_{X=b}/\hat{Y}_{X=a}$ a risk ratio, and the corresponding odds comparison an odds ratio.
- The `marginaleffects` package computes all of these quantities using a consistent interface.
### First steps: risk difference with a binary treatment {#sec-comparisons_first_steps}
- The first estimand considered is the risk difference for a change in binary treatment: the expected change in outcome when `incentive` changes from 0 to 1.
- Counterfactual comparisons are *conditional* quantities: except in the simplest cases, they depend on the values of all the predictors in the model.
- Each individual in a dataset may be associated with a different counterfactual comparison.
- When computing a comparison, the analyst must define values for the focal predictor and all other covariates.
- The basic procedure is: create counterfactual grids (treatment vs. control), generate predictions for each, and take the difference.
- The `comparisons()` function from `marginaleffects` automates this procedure and also returns standard errors and test statistics.
- For example, for a participant aged 18-35 living at distance 2, moving from control to treatment increases the predicted probability of the outcome by several percentage points.
- Interpreting a counterfactual comparison as a causal effect requires strong assumptions; without them, the comparison is a descriptive measure of the model's behavior. @sec-gcomputation discusses these assumptions.
### Comparison functions {#sec-comparisons_functions}
- By default, comparisons measure effects via differences in predicted outcomes.
- The `comparison` argument in `comparisons()` allows alternative functions: `"ratio"`, `"lift"`, or custom functions.
- Setting `comparison="ratio"` computes the ratio of predicted outcomes under treatment vs. control.
- The `hypothesis` argument can be set (e.g., `hypothesis=1`) to test against a null hypothesis that the ratio equals 1.
- Setting `comparison="lift"` computes the relative change: $(\hat{Y}_{hi} - \hat{Y}_{lo})/\hat{Y}_{lo}$.
- The `comparison` argument also accepts arbitrary user-defined functions, taking `hi` (treatment) and `lo` (control) prediction vectors.
- This allows fully customized comparisons, such as a log odds ratio based on average predictions.
- Odds ratios are non-collapsible, so the chapter computes the log odds ratio of the averages rather than the average log odds ratio.
## Predictors {#sec-comparisons_predictors}
- Predictors divide into two categories: focal variables (whose effect we want to quantify) and adjustment (control) variables (included for flexibility, fit, or confounding).
- The effect of an adjustment variable is not of inherent interest in a counterfactual analysis; interpreting adjustment variable parameters as effects is generally not recommended (Table 2 fallacy).
- Counterfactual comparisons are conditional quantities that depend on the values of all predictors, so analysts must decide where in the predictor space to evaluate them.
- The `variables` argument identifies the focal predictor(s), and the `newdata` argument (or `datagrid()`) defines the grid of covariate values.
- Different predictor types (binary, categorical, numeric) require different specifications of the contrast.
- Cross-comparisons allow assessing the joint effect of manipulating two or more predictors simultaneously using the `cross` argument.
### Focal variables {#sec-comparisons_focal}
- For binary predictors, `marginaleffects` returns the difference in predicted outcome for a change from 0 to 1 by default; the direction can be reversed using list syntax (e.g., `variables = list("incentive" = c(1, 0))`).
- For categorical predictors with multiple levels, the default compares each level to the reference (first) category; the `variables` argument supports specific pairs (e.g., `c("18 to 35", ">35")`) or sequential comparisons.
- For numeric predictors, the default is a 1-unit increase from the value in the predictor grid; other options include a specific number of units, `"sd"`, `"iqr"`, `"minmax"`, or two specific values.
- Cross-comparisons assess the combined effect of changing two predictors simultaneously using `cross = TRUE`.
- In a cross-comparison, the result shows the joint effect of, e.g., changing both `distance` by 1 unit and `incentive` from 0 to 1.
- For pedagogical purposes, the chapter treats each predictor (`incentive`, `agecat`, `distance`) as focal in turn, though in practice only one or two focal predictors per model are typical.
### Adjustment variables
- The researcher is not interested in changes to adjustment variables themselves, but must still define their values because comparisons depend on the full predictor grid.
- By default, `comparisons()` returns estimates for every row of the original dataset (empirical distribution).
- If `variables` is not specified, `comparisons()` computes differences for all variables, yielding a large output.
- Since the output is a data frame, unit-level risk differences can be plotted to reveal heterogeneity: for some individuals, the treatment effect may be much larger or smaller than for others.
- Interesting grids: supply a data frame to `newdata` (or use `datagrid()`) to estimate comparisons for specific profiles of interest, e.g., `datagrid(agecat = unique, distance = mean)`.
- Representative grids: using `newdata = "mean"` creates a synthetic "average" individual; this is fast but the interpretation is ambiguous since no real individual may be perfectly average on all dimensions.
- A "comparison at the mean" can differ from an "average comparison."
- Balanced grids (`newdata = "balanced"`) include all unique combinations of categorical variables while holding numeric variables at their means, useful in experimental contexts.
## Aggregation {#sec-comparisons_aggregation}
- The default behavior of `comparisons()` is to produce unit-level estimates; to marginalize across units, use `avg_comparisons()`.
- The average treatment effect (ATE) is the expected difference in outcomes under treatment vs. control, with expectation taken over the distribution of adjustment variables: $E[Y_{X=1} - Y_{X=0}]$.
- To compute an average comparison, the procedure is: (1) predict under treatment for all rows, (2) predict under control for all rows, (3) take differences, (4) average.
- `avg_comparisons()` returns the same result as computing unit-level comparisons and taking the mean.
- The `by` argument enables subgroup-specific average comparisons, e.g., average risk difference by age category.
- The `newdata` argument can restrict the average to specific subsets, e.g., the treated group only, yielding an ATT-like quantity.
- The chapter distinguishes three related quantities: average predictions, average counterfactual predictions, and average counterfactual comparisons, using a Palmer Penguins example.
### Average predictions vs. average comparisons {#sec-comparisons_average_predictions_vs_comparisons}
- Average predictions are calculated over the observed distribution of covariates within each subset; differences between groups reflect both the focal variable and covariate differences.
- Average counterfactual predictions replicate the full dataset for each level of the focal variable, holding the covariate distribution identical across groups; they are *ceteris paribus* quantities.
- Because counterfactual predictions hold covariates constant, the gaps between groups are typically smaller than the gaps between raw average predictions.
- Average counterfactual comparisons measure the differences between counterfactual predictions, giving the estimated effect of the focal variable while controlling for covariates.
- `avg_comparisons()` computes these directly, or they can be obtained as differences between `avg_predictions()` estimates with the `variables` and `by` arguments.
- This illustrates how counterfactual comparisons enable "all else equal" analyses, controlling for the fact that groups may differ on covariates.
## Uncertainty {#sec-comparisons_uncertainty}
- Standard errors for contrasts are computed by default using the delta method and the classical variance-covariance matrix from the modeling software.
- Robust standard errors (e.g., heteroskedasticity-consistent or clustered) can be obtained by setting the `vcov` argument (e.g., `vcov = "HC3"` or `vcov = ~village`).
- The `inferences()` function supports bootstrap or simulation-based estimates of uncertainty.
- The `modelsummary` package can display estimates with different uncertainty approaches side-by-side in a single table.
- The `statistic` argument in `modelsummary` can switch from standard errors to confidence intervals.
- The `fmt` argument controls the number of displayed digits, and `gof_omit` or `gof_map` controls goodness-of-fit statistics shown.
## Test {#sec-comparisons_test}
- Hypothesis tests on counterfactual comparisons allow comparing subgroup-specific estimates to one another.
- The motivating question: does the incentive treatment have a bigger effect for older or younger participants?
- Use `avg_comparisons()` with `by = "agecat"` to obtain subgroup-specific average risk differences.
- Even when point estimates differ numerically between subgroups, the difference may not be statistically significant.
- The `hypothesis` argument expresses the test as a string formula (e.g., `"b1 - b3 = 0"` in R, `"b0 - b2 = 0"` in Python) where `b1`/`b0` refer to row indices.
- A large $p$ value means we cannot reject the null that the treatment effect is the same across age brackets; the subgroup estimates are not statistically distinguishable.
- This approach generalizes to any linear or non-linear hypothesis about comparisons, as described in @sec-hypothesis.
## Visualization {#sec-comparisons_visualization}
- Data analysts can visualize counterfactual comparisons with `plot_comparisons()`, which uses the same arguments as `comparisons()` and `avg_comparisons()`.
- Unit-level comparisons can be plotted as histograms to show the distribution of treatment effects across individuals.
- `plot_comparisons()` can present marginal comparisons (average effects by subgroup) or conditional comparisons (effects as a function of a continuous predictor).
### Marginal comparisons
- Use `plot_comparisons()` with the `by` argument to visualize average treatment effects across subgroups (e.g., average risk difference by age category).
- The resulting plot may show that point estimates differ across subgroups, but wide and overlapping confidence intervals suggest the differences are not statistically significant.
- This provides a visual complement to the formal hypothesis tests in the Test section.
- The syntax mirrors `avg_comparisons()` but produces a graphical output.
- Labels and axes can be customized with standard `ggplot2` functions like `labs()`.
### Conditional comparisons
- When a predictor of interest is continuous or multiple predictors are involved, conditional comparisons improve interpretability.
- The `condition` argument of `plot_comparisons()` specifies which predictors to display on the x-axis and which to hold at representative values or facet by.
- For example, plotting the risk difference for `incentive` as a function of `distance` shows how treatment effects vary with distance from the test center.
- Adding a second variable (e.g., `agecat`) to `condition` creates separate curves or facets for each subgroup.
- These plots reveal that the estimated effect of incentive is smaller for individuals closest to the test center, and that older participants may react less strongly.
- @sec-interactions shows how to design and interpret formal tests of treatment effect heterogeneity using interactions and polynomials.
# Examples
- The examples below are drawn from the original chapter and illustrate representative workflows.
## Example 1: Risk difference for a specific individual
```r
library(marginaleffects)
dat <- get_dataset("thornton")
mod <- glm(outcome ~ incentive * (agecat + distance),
data = dat, family = binomial)
# Define a profile of interest
grid <- data.frame(distance = 2, agecat = "18 to 35", incentive = 1)
# Compute the risk difference for this individual
comparisons(mod, variables = "incentive", newdata = grid)
```
## Example 2: Average treatment effect by subgroup
```r
# Average risk difference for each age category
avg_comparisons(mod, variables = "incentive", by = "agecat")
# Hypothesis test: is the effect different for youngest vs. oldest?
avg_comparisons(mod,
hypothesis = "b1 - b3 = 0",
variables = "incentive",
by = "agecat")
```
## Example 3: Conditional comparison plot
```r
library(ggplot2)
# How does the treatment effect vary with distance?
plot_comparisons(mod,
variables = "incentive",
condition = c("distance", "agecat")) +
labs(y = "Conditional risk difference")
```
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `slopes`, `avg_slopes`, `plot_slopes`, `datagrid`, `inferences` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `slopes`, `avg_slopes`, `plot_slopes`, `datagrid`, `inferences` for this chapter.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
## Summary
This chapter defined a "slope" as the partial derivative of the regression equation with respect to a predictor of interest. It is a measure of association between two variables, or of the effect of one variable on another, holding other predictors constant. The `slopes()` function from the `marginaleffects` package computes slopes for a wide range of models. `avg_slopes()` aggregates slopes across units or groups. `plot_slopes()` displays slopes visually.
To clearly define slopes and attendant tests, analysts must make five decisions.
First, the *Quantity*.
- A slope is always computed with respect to a focal predictor, whose effect on (or association with) the outcome we wish to estimate. In `marginaleffects` functions, the focal variable is specified using the `variables` argument.
- A slope can roughly be interpreted as the effect of a one-unit change in the focal predictor on the predicted outcome. However, this interpretation is a linear approximation, valid only in a small neighborhood of the predictors.
Second, the *Predictors*.
- Slopes are conditional quantities, meaning that they will typically vary based on the values of all predictors in a model. Every row of a dataset has its own slope.
- Analysts can compute slopes for different combinations of predictor values---or grids: empirical, interesting, representative, balanced, or counterfactual.
- The predictor grid is defined by the `newdata` argument and the `datagrid()` function.
Third, the *Aggregation*.
- To simplify the presentation of results, analysts can report average slopes. Different aggregation schemes are available:
- Unit-level slopes (no aggregation)
- Average slopes
- Average slopes by subgroup
- Weighted average of slopes
- Slopes can be aggregated using the `avg_slopes()` function and the `by` argument.
Fourth, the *Uncertainty*.
- In `marginaleffects`, the `vcov` argument allows analysts to report classical, robust, or clustered standard errors for slopes.
- The `inferences()` function can compute uncertainty intervals via bootstrapping or simulation-based inference.
Fifth, the *Test*.
- A null hypothesis test evaluates whether a slope (or a function of slopes) is significantly different from a null value. For example, we may use a null hypothesis test to check if treatment effects are equal in subgroups of the sample. Null hypothesis tests are conducted using the `hypothesis` argument.
- An equivalence test evaluates whether a slope (or a function of slopes) is similar to a reference value. Equivalence tests are conducted using the `equivalence` argument.
# Slopes {#sec-slopes}
- A slope measures how the predicted value of the outcome $Y$ responds to changes in a focal predictor $X$, when we hold other covariates at fixed values. It is often the main quantity of interest when a researcher wants to estimate an effect or the strength of association between two variables.
- Slopes belong to the same toolbox as counterfactual comparisons: both help us answer what would happen to a predicted outcome if one of the predictors were slightly different.
- In this book, "slope" and "marginal effect" are used interchangeably to mean: partial derivative of the regression equation with respect to a predictor of interest. Other terms used in the literature include "trend," "velocity," and "partial effect."
- In a simple linear model $Y = \beta_0 + \beta_1 X + \varepsilon$, the slope with respect to $X$ is $\frac{\partial Y}{\partial X} = \beta_1$. This explains why some analysts refer to regression coefficients as slopes.
- The interpretation of a slope as the effect of a one-unit change in $X$ is a linear approximation, valid only for small changes in $X$ in a small neighborhood of the predictors. The approximation may not be good when the regression function is non-linear.
- Since slopes are defined as derivatives (infinitesimal changes), they can only be constructed for continuous numeric predictors. Analysts interested in the effect of a change in a categorical predictor should use counterfactual comparisons instead.
- Slopes are conditional quantities: they typically depend on the value of $X$ and on the values of all other predictors in the model. Every row of a dataset or grid has its own slope.
- The rest of the chapter proceeds through the five questions of the conceptual framework: (1) quantity, (2) predictors, (3) aggregation, (4) uncertainty, and (5) test.
## Quantity
- A slope characterizes the strength and direction of association between a predictor and an outcome, holding other covariates constant.
- The slope of a curve tells us whether the curve is increasing, decreasing, or flat as we move along the x-axis.
- By examining the values of derivatives at any given point, we can determine precisely where the corresponding functions are rising, falling, or flat.
- In non-linear models (e.g., logistic regression), the slope is far from constant---it depends on the baseline value of the predictor.
- The derivative of a function precisely characterizes both the strength and direction of association between a predictor and an outcome at any given point in the predictor space.
- The `plot_predictions()` and `plot_slopes()` functions can be used to visualize the outcome function and its derivative side by side.
### Slopes of simple functions
- For the linear function $Y = -1 + 0.5X$, the derivative is constant: $\frac{\partial Y}{\partial X} = 0.5$. The association between $X$ and $Y$ is always positive and of constant magnitude.
- For the quadratic function $Y = X^2$, the derivative is $\frac{\partial Y}{\partial X} = 2X$. When $X < 0$, the relationship is negative; when $X = 0$, the slope is zero; when $X > 0$, the relationship is positive. The sign and strength of the relationship depend on the baseline value of $X$.
- For $Y = \cos(X)$, the derivative is $\frac{\partial Y}{\partial X} = -\sin(X)$. Whenever $-\sin(X)$ is negative, $\cos(X)$ points downward; when $-\sin(X) = 0$, the curve is flat; when $-\sin(X) > 0$, it points upward.
- These three examples show that the derivative precisely characterizes both the strength and direction of association between a predictor and an outcome.
- The sign of the derivative tells us if $Y$ is increasing or decreasing, and the magnitude tells us how fast.
- The heterogeneity in slopes across the predictor space is a key insight for studying interactions and non-linearity in regression models.
### Slope of a logistic function
- Consider a logistic regression: $\Pr(Y=1) = g(\beta_1 + \beta_2 X)$, where $g$ is the logistic function. With true parameters $\beta_1 = -1$ and $\beta_2 = 0.5$, we can simulate data and fit a model with `glm()`.
- The key insight is that the slope of the logistic function is far from constant. Increasing $X$ has a different effect on $\Pr(Y=1)$ depending on the baseline position on the horizontal axis.
- When $X$ is small or large, the prediction curve is flat and the derivative is close to zero. At intermediate values, the curve is steep and the derivative is large and positive.
- Taking the derivative of the logistic equation using the chain rule: $\frac{\partial \Pr(Y=1)}{\partial X} = \beta_2 \cdot g'(\beta_1 + \beta_2 X)$, where $g'$ is the logistic density function.
- To evaluate a slope, one must explicitly state the baseline values of the predictors. The slope will differ at different values of $X$.
- The `slopes()` function computes these derivatives. The `variables` argument specifies the focal predictor, and `newdata` specifies the predictor values at which to evaluate the slope.
- The `plot_predictions()` and `plot_slopes()` functions visualize the outcome function and its derivative, and the `patchwork` package can combine them vertically.
## Predictors {#sec-slopes_predictors}
- Slopes answer: how does the predicted outcome $\hat{Y}$ change when the focal variable $X$ increases by a small amount and the adjustment variables $Z_1, Z_2, \ldots, Z_n$ are held at specific values?
- Answering this requires selecting both the focal variable and the values of the adjustment variables where we want to evaluate the slope.
- The focal variable is specified via the `variables` argument; the adjustment variables are set via the `newdata` argument and the `datagrid()` function.
- Slopes are conditional quantities: every predictor profile (combination of predictor values) is associated with its own slope.
- The `slopes()` function returns unit-level marginal effects by default---one slope per row in the dataset.
- The `newdata="mean"` shortcut computes a "marginal effect at the mean" for an individual whose characteristics are exactly average or modal on all predictors.
- Choosing to report slopes at the mean vs. at user-specified values can yield quite different results, emphasizing the crucial importance of grid definition.
### Focal variable
- The focal variable is the predictor of interest---the variable whose association with (or effect on) $Y$ we wish to estimate.
- It is the predictor in the denominator of the partial derivative: $\frac{\partial Y}{\partial X}$.
- The `slopes()` function accepts a `variables` argument to specify the focal predictor.
- For example, `slopes(mod, variables = "distance")` estimates the partial derivative of the outcome with respect to `distance`.
- A one-unit increase in the focal predictor is associated with a change in the predicted outcome equal to the slope, but this is a linear approximation valid only for small changes.
### Adjustment variables
- Slopes are conditional quantities: they typically depend on the values of all variables on the right-hand side of a regression equation.
- Every predictor profile (combination of predictor values) is associated with its own slope; every row in a grid has its own slope.
- The "interesting" or "user-specified" grid collects combinations of predictor values that hold particular scientific or domain-specific interest.
- The `datagrid()` helper function specifies a grid of predictor values, which is passed to the `newdata` argument of `slopes()`.
- The function then returns a "slope at user-specified values" or "marginal effect at interesting values."
- Instead of manually specifying predictor values, `newdata="mean"` computes slopes for an individual with average or modal characteristics. But the "mean" individual may not be realistic or substantively interesting.
- Analysts can also obtain unit-level marginal effects (one per observation) as the default output of `slopes()`.
## Aggregation {#sec-slopes_aggregation}
- A dataset with one marginal effect estimate per observation is unwieldy and difficult to interpret. Many analysts prefer to report the "average marginal effect" (or "average slope"), the average of all unit-level estimates.
- The `avg_slopes()` function conveniently computes average slopes in one step.
- There is a nuanced distinction between the "marginal effect at the mean" and the "average marginal effect." The former is based on a single individual with exactly average characteristics; the latter averages slopes across all observed data points.
- These two options are not always equivalent; they can yield numerically and substantively different results.
- The marginal effect at the mean may be useful under computational constraints. The average marginal effect is useful when the dataset represents the population distribution of predictors.
- The `by` argument computes "conditional average marginal effects"---average slopes by subgroup---to explore heterogeneity in the association between predictors and the outcome.
- For example, using `avg_slopes(mod, variables = "distance", by = "incentive")` reveals whether the association between `distance` and `outcome` differs by incentive group.
## Uncertainty {#sec-slopes_uncertainty}
- Uncertainty around slopes can be estimated using various strategies: classical or robust standard errors, bootstrapping, simulation-based inference, etc.
- The `vcov` argument specifies the variance-covariance strategy (e.g., `vcov = ~village` for clustered standard errors).
- The `conf_level` argument controls the size of confidence intervals.
- The `inferences()` function provides bootstrapping and simulation-based inference (e.g., `inferences(method = "boot")`).
- Different uncertainty quantification strategies yield the same point estimates but different confidence intervals.
- For example, `avg_slopes(mod, variables = "distance", vcov = ~village)` clusters standard errors by village, while piping to `inferences(method = "boot")` applies a non-parametric bootstrap.
## Test {#sec-slopes_test}
- The hypothesis testing approach from earlier chapters is directly applicable to slopes, just as it was to model coefficients, predictions, and counterfactual comparisons.
- The `hypothesis` argument accepts an equation-like string specifying the null hypothesis, e.g., `hypothesis = "b1 - b2 = 0"`.
- In the running example, the average slope of `outcome` with respect to `distance` appears different across `incentive` subgroups: distance seems more discouraging to people who do not receive an incentive.
- To formally test this, we compute `avg_slopes(mod, variables = "distance", by = "incentive", hypothesis = "b1 - b2 = 0")`.
- If the p-value is large, we cannot reject the null hypothesis of homogeneity in the effect of distance across subgroups.
- This framework generalizes: analysts can conduct linear or non-linear hypothesis tests on any slope or function of slopes.
## Visualization {#sec-slopes_visualization}
- Predictions give the level of an expected outcome for given predictor values; slopes capture how the expected outcome changes in response to a change in a focal variable, holding adjustment variables constant.
- The top panel of a combined figure shows model-based predictions at different values of `distance`; the bottom panel shows the corresponding slope of the predicted outcome with respect to `distance`.
- When the slope is negative, the prediction curve declines. When the slope is zero, the prediction curve is flat. This correspondence helps interpret the model.
- The `plot_predictions()` and `plot_slopes()` functions draw these panels, and the `patchwork` `/` operator combines them vertically.
- `plot_slopes()` can display slopes based on multiple conditions (e.g., `condition = c("distance", "incentive")`) to show how the association varies across subgroups.
- The `by` argument in `plot_slopes()` computes marginal (average) slopes by subgroup, useful for visualizing the results explored in hypothesis tests.
- In the running example, the negative association between `distance` and `outcome` seems weaker when `incentive=1`, but wide confidence intervals prevent rejecting the null of equal slopes.
# Examples
The examples below are drawn from the original chapter and illustrate representative workflows.
## Example 1: Compute slopes at user-specified predictor values
```r
library(marginaleffects)
dat <- get_dataset("thornton")
mod <- glm(outcome ~ incentive * distance * I(distance^2),
data = dat, family = binomial)
# Slope of outcome w.r.t. distance for a treated individual at distance=1
slopes(mod,
variables = "distance",
newdata = datagrid(incentive = 1, distance = 1))
```
## Example 2: Average slopes by subgroup with hypothesis test
```r
# Average slope of outcome w.r.t. distance, by incentive group
avg_slopes(mod, variables = "distance", by = "incentive")
# Test whether the two subgroup slopes are statistically distinguishable
avg_slopes(mod,
variables = "distance",
by = "incentive",
hypothesis = "b1 - b2 = 0")
```
## Example 3: Visualize predictions and slopes
```r
library(ggplot2)
library(patchwork)
p1 <- plot_predictions(mod, condition = "distance") +
labs(y = "Predicted Pr(Outcome=1)")
p2 <- plot_slopes(mod, variables = "distance", condition = "distance") +
geom_hline(yintercept = 0, linetype = "dotted") +
labs(y = "dY/dX")
p1 / p2
```
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `avg_comparisons`, `avg_predictions`, `predictions`, `comparisons`, `transform`, `subset`, `inferences` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `avg_comparisons`, `avg_predictions`, `predictions`, `comparisons` for this chapter. Also see `polars` functions `with_columns`, `lit`, `alias`, `filter`, `select`.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
# Causal inference with G-computation {#sec-gcomputation}
- Randomized experiments are the gold standard for causal inference because random assignment ensures groups have similar background characteristics on average.
- When randomized experiments are impractical or unethical, analysts must rely on observational data, where confounding variables can bias treatment effect estimates.
- G-computation (also called the Parametric G-Formula) is a method for drawing causal inference from observational data developed by Robins and colleagues.
- The procedure has three steps: (1) fit a statistical model that controls for confounders, (2) use the model to impute (predict) outcomes under alternative treatment scenarios, and (3) compare counterfactual predictions to estimate the treatment effect.
- G-computation estimates are equivalent to the counterfactual predictions and comparisons discussed in earlier chapters on predictions and comparisons.
- G-computation is closely related to Inverse Probability Weighting (IPW): both impose similar identification assumptions, but G-computation models the outcome while IPW models the treatment assignment mechanism.
- G-computation and IPW can be combined into a "doubly-robust" estimator with desirable statistical properties.
- The chapter targets four estimands: the average treatment effect (ATE), average treatment effect on the treated (ATT), average treatment effect on the untreated (ATU), and conditional average treatment effect (CATE).
## Treatment effects: ATE, ATT, ATU {#sec-gcomputation_treatment_effects}
- The goal is to estimate the effect of a treatment $D$ on an outcome $Y$ using the potential outcomes framework.
- $D_i=1$ indicates individual $i$ is assigned to treatment; $D_i=0$ indicates assignment to control.
- Potential outcomes $Y_i^1$ and $Y_i^0$ represent the outcomes that would occur under treatment and control, respectively; only one can ever be observed for a given individual.
- The Individual Treatment Effect (ITE) is $Y_i^1 - Y_i^0$, but it is impossible to compute because we observe only one potential outcome per individual -- this is the fundamental problem of causal inference.
- To circumvent this, we focus on three aggregate estimands: ATE, ATT, and ATU, which are related but invite different interpretations and impose different assumptions.
- The ATE, ATT, and ATU each answer distinct policy questions about whether a treatment should be implemented universally, continued for current recipients, or extended to non-recipients.
### Interpretation
- The three estimands are defined as: ATE = $E[Y_i^1 - Y_i^0]$, ATT = $E[Y_i^1 - Y_i^0 \mid D_i=1]$, ATU = $E[Y_i^1 - Y_i^0 \mid D_i=0]$.
- The ATE estimates the average treatment effect across the entire study population and answers: should the treatment be implemented universally?
- The ATT estimates the average treatment effect among those who actually received treatment and answers: should the treatment be withheld from current recipients?
- The ATU estimates the average treatment effect among those who did not receive treatment and answers: should the treatment be extended to non-recipients?
- These three quantities can differ substantially, underscoring the importance of clearly defining the target estimand before estimation.
### Assumptions {#sec-gcomputation_assumptions}
- Estimating the ATE, ATT, or ATU via G-computation requires four assumptions of varying stringency.
- **Conditional exchangeability** (no unmeasured confounding): potential outcomes must be independent of treatment assignment, conditional on control variables $Z_i$, written as $Y_i^1, Y_i^0 \perp D_i \mid Z_i$. This is violated when treatment assignment depends on potential outcomes (e.g., self-selection into a training program or a doctor prescribing based on expected benefit).
- For the ATE, conditional exchangeability must hold across the entire population. For the ATT, it can be relaxed: since $Y_i^1$ is observed for the treated, we only need assumptions about the unobserved $Y_i^0$. Analogously, the ATU requires exchangeability only in the treatment condition.
- **Positivity**: every individual must have a non-zero probability of being in each treatment arm, $0 < P(D_i=1 \mid Z_i) < 1$. For the ATT, this relaxes to $P(D_i=1 \mid Z_i) < 1$; for the ATU, to $P(D_i=1 \mid Z_i) > 0$.
- **Consistency**: the intervention must be well-defined with no ambiguity about what constitutes "treatment" and "control" (e.g., same drug, dosage, and conditions for all treated individuals).
- **Non-interference** (part of SUTVA): potential outcomes for one participant must not be affected by the treatment assignments of other individuals -- no contagion or externalities across participants.
- When all four assumptions hold, we can estimate treatment effects via G-computation using the three-step procedure: model, impute, and compare.
### Model
- The chapter illustrates G-computation using a study by Imbens, Rubin, and Sacerdote (2001) on the effect of winning large lottery prizes on labor earnings.
- Treatment ($D$): binary indicator for winning a big lottery prize. Outcome ($Y$): average labor earnings over six subsequent years.
- Winning numbers are random, but the probability of winning depends on the number of tickets purchased ($L$), and socio-demographic characteristics ($Z$) influence both ticket purchases and earnings.
- The dataset includes 437 respondents; the analysis compares big-prize winners to non-winners (excluding small-prize winners).
- The first G-computation step is to fit an outcome model that controls for confounders to satisfy conditional exchangeability.
- To satisfy the backdoor criterion, the model must control for the number of tickets ($L$) purchased and/or the covariates ($Z$) linking treatment to outcome.
- The model can be simple or flexible: linear, with interactions, polynomials, or splines; it can use a minimal or expanded set of controls.
- In the example, a linear regression interacts `win_big` with `tickets`, gender, age, employment, education, college, year, and three years of pre-lottery earnings.
### Impute
- The second G-computation step is to impute (predict) the outcome for each individual under different hypothetical treatment regimes.
- Two copies of the dataset are created: one with `win_big` set to 0 for all individuals, and one with `win_big` set to 1 for all individuals.
- In R, counterfactual datasets are created using `transform(dat, win_big = 0)` and `transform(dat, win_big = 1)`.
- In Python, counterfactual datasets are created using `dat.with_columns(pl.lit(0).alias('win_big'))` and the analogous expression for 1.
- The `predictions()` function from `marginaleffects` generates predicted outcomes for each individual under both counterfactual scenarios.
- Inspecting individual-level predictions shows that the model predicts different earnings for the same person depending on treatment status -- for example, lower earnings under the treatment (winning) scenario.
### Compare
- The third G-computation step is to aggregate individual-level counterfactual predictions to estimate the treatment effect.
- The ATE is obtained by computing the difference in the mean predicted outcomes between the treatment and control counterfactual datasets.
- The `avg_predictions()` function with the `variables` and `by` arguments provides the same estimates as manual calculation, but with standard errors and confidence intervals.
- The `avg_comparisons()` function directly computes the ATE as the difference between counterfactual means, providing a single estimate with uncertainty measures.
- The ATT is computed using the same approach but restricting `newdata` to the subset of treated individuals (e.g., `newdata = subset(win_big == 1)` in R or `newdata=dat.filter(pl.col('win_big') == 1)` in Python).
- The ATU is computed analogously by restricting `newdata` to untreated individuals.
- In the lottery example, the ATT is larger than the ATU, suggesting that people who buy more tickets (and are therefore more likely to win) experience a stronger decrease in labor earnings after winning.
- This asymmetry underscores the importance of clearly defining one's estimand before estimation and interpretation.
- Standard errors for G-computation estimates involve subtle issues: treating covariates as fixed rather than sampled may yield inadequate coverage. The bootstrap (e.g., via the `inferences()` function) is a common alternative; analytic unconditional variance expressions are also available.
## Conditional treatment effects: CATE {#sec-gcomputation_cate}
- The conditional average treatment effect (CATE) characterizes how the treatment effect varies across subgroups, defined as $E[Y_i^1 - Y_i^0 \mid X_i = x]$.
- Unlike the ATE, which averages across the entire population, the CATE conditions on specific values of a covariate $X_i$ to reveal heterogeneity in treatment effects.
- The conditioning variable can be any discrete variable such as education level or employment status.
- In the lottery example, the CATE is estimated by employment status using `avg_comparisons(mod, variables = "win_big", by = "work")`.
- For initially unemployed individuals (`work=0`), the estimated effect of winning on labor earnings is close to zero and not statistically significant.
- For employed individuals (`work=1`), winning a big prize significantly reduces labor earnings, consistent with the theory that a wealth windfall reduces work incentives for those with labor market attachment.
- CATEs can be estimated with additional conditioning variables by adding more categorical variables to the `by` argument of `avg_comparisons()`, though this reduces subgroup sample sizes.
- G-computation provides a useful framework for estimating causal effects from observational data by modeling, imputing, and comparing potential outcomes to target the ATE, ATT, ATU, and CATE while accounting for confounders.
# Examples
- The examples below are drawn from the original chapter and illustrate representative workflows.
## Example 1: Estimating the ATE with G-computation
```r
library(marginaleffects)
dat = get_dataset("lottery")
dat = subset(dat, win_big == 1 | win == 0)
# Step 1: Model -- fit outcome regression controlling for confounders
mod = lm(
earnings_post_avg ~ win_big * (
tickets + man + work + age + education + college + year +
earnings_pre_1 + earnings_pre_2 + earnings_pre_3),
data = dat)
# Step 2: Impute -- predict under counterfactual treatment regimes
d0 = transform(dat, win_big = 0)
d1 = transform(dat, win_big = 1)
p0 = predictions(mod, newdata = d0)
p1 = predictions(mod, newdata = d1)
# Step 3: Compare -- estimate the ATE
avg_comparisons(mod, variables = "win_big", newdata = dat)
```
## Example 2: ATT and ATU using subset restrictions
```r
# ATT: restrict to treated individuals
avg_comparisons(mod, variables = "win_big",
newdata = subset(win_big == 1))
# ATU: restrict to untreated individuals
avg_comparisons(mod, variables = "win_big",
newdata = subset(win_big == 0))
```
## Example 3: Conditional Average Treatment Effect (CATE)
```r
# CATE by employment status
avg_comparisons(mod, variables = "win_big", by = "work")
```
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `avg_comparisons`, `plot_predictions`, `get_dataset`, `lm`, `coef` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `avg_comparisons`, `ols`, `to_pandas`, `fit`, `get_dataset`, `plot_predictions` for this chapter.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
# Experiments {#sec-experiments}
- The analysis of experiments is a common use case for the `marginaleffects` package, which provides a toolkit for estimating treatment effects, interpreting interactions, and visualizing results across experimental conditions.
- This chapter discusses two applications: covariate adjustment in a two-arm experiment, and the interpretation of results from a 2-by-2 factorial experiment.
- The marginaleffects.com website hosts additional tutorials for other experimental designs.
- Both applications rely on the `avg_comparisons()` function to estimate average treatment effects using G-computation, producing results that are agnostic to the choice of model (e.g., LPM vs. GLM).
- All code examples are shown in both R and Python, using the same workflow and function interface.
- Data for the first application come from @Tho2008: a binary outcome recording whether individuals traveled to a clinic to learn their HIV status, with a randomized financial incentive as treatment.
- Data for the second application use a simulated dataset with 32 observations, a numeric outcome $Y$, and two binary treatments $T_a$ and $T_b$.
- The `marginaleffects` workflow handles uncertainty quantification, including heteroskedasticity-consistent and robust standard errors via the `vcov` argument.
## Regression adjustment {#sec-experiments_adjustment}
- The first application is a simple two-arm experiment where participants are randomly assigned to treatment or control.
- The goal is to estimate the average treatment effect (ATE) -- the expected difference in outcomes between treatment and control -- while adjusting for covariates.
- Since the incentive treatment was randomized, the ATE can be estimated via linear regression on the binary outcome, known as the linear probability model (LPM).
- The coefficient on the `incentive` variable shows the difference in predicted probability of seeking test results between treatment and control groups.
- The same result is obtained via G-computation using `avg_comparisons()`, which has the advantage of easily supporting robust standard errors through the `vcov` argument (e.g., `vcov = "HC2"`).
- The `marginaleffects` package implements post-estimation transformations that are essentially agnostic with respect to the choice of model; the workflow would remain unchanged with a GLM instead of an LPM.
- While covariate adjustment is not strictly necessary for an unbiased ATE when treatment is randomized, including control variables can improve precision by reducing unexplained variance.
- However, @Fre2008 warns that naively inserting control variables in additive fashion can introduce small-sample bias and degrade asymptotic precision.
- @Lin2013 recommends a simple solution: interact all covariates with the treatment indicator and report heteroskedasticity-robust standard errors.
- Interpreting the coefficients of a model with multiple interactions is not straightforward, but calling `avg_comparisons()` produces the covariate-adjusted ATE directly without requiring de-meaned control variables.
## Factorial experiments {#sec-experiments_factorial}
- A factorial experiment is a study design that assesses the effects of two or more randomized treatments simultaneously, with each treatment having multiple levels.
- A common example is the 2-by-2 design, where two binary variables are randomized simultaneously and independently, enabling evaluation of each treatment's effect and their interaction.
- Factorial experiments are used in medicine (drug interactions), plant physiology (temperature and humidity effects on photosynthesis), and business (advertising strategies and price points).
- The chapter uses a simulated dataset with 32 observations, a numeric outcome $Y$, and two binary treatments $T_a \in \{0,1\}$ and $T_b \in \{0,1\}$, fit with a linear model including a multiplicative interaction term.
- Predictions for each combination of $T_a$ and $T_b$ can be visualized using `plot_predictions()`.
- Treatment effects can be computed by hand as sums of coefficients (e.g., the effect of $T_a$ holding $T_b=0$ equals $\hat{\beta}_2$; the cross-contrast equals $\hat{\beta}_2 + \hat{\beta}_3 + \hat{\beta}_4$), but this becomes tedious and computing standard errors is non-trivial.
- The `avg_comparisons()` function automates these calculations: use the `variables` argument to specify the treatment of interest, and `newdata = subset(Tb == 0)` (R) or `newdata = dat.filter(pl.col("Tb") == 0)` (Python) to condition on specific levels of the other treatment.
- To estimate a cross-contrast (the effect of changing both $T_a$ and $T_b$ simultaneously from 0 to 1), set `cross = TRUE` in `avg_comparisons()`.
- To assess whether treatments interact (i.e., whether the effect of $T_a$ depends on the value of $T_b$), estimate the effect of $T_a$ by subgroups of $T_b$ using the `by` argument, then use the `hypothesis` argument to test if the difference between subgroup estimates is significant.
- In the example, the difference between the estimated effects of $T_a$ at different values of $T_b$ is not statistically significant, so we cannot reject the null hypothesis that the effect of $T_a$ is the same regardless of $T_b$.
# Examples
- The examples below are drawn from the original chapter and illustrate representative workflows.
## Example 1: Covariate-adjusted ATE with robust standard errors
```r
library(marginaleffects)
dat <- get_dataset("thornton")
mod <- lm(outcome ~ incentive * (age + distance + hiv2004), data = dat)
avg_comparisons(mod, variables = "incentive", vcov = "HC2")
```
```python
from marginaleffects import *
from statsmodels.formula.api import ols
dat = get_dataset("thornton")
mod = ols("outcome ~ incentive * (age + distance + hiv2004)",
data=dat.to_pandas()).fit()
avg_comparisons(mod, variables = "incentive", vcov = "HC2")
```
## Example 2: Factorial cross-contrast
```r
library(marginaleffects)
dat <- get_dataset("factorial_01")
mod <- lm(Y ~ Ta + Tb + Ta:Tb, data = dat)
avg_comparisons(mod, variables = c("Ta", "Tb"), cross = TRUE)
```
```python
dat = get_dataset("factorial_01")
mod = ols("Y ~ Ta + Tb + Ta:Tb", data=dat.to_pandas()).fit()
avg_comparisons(mod, variables=["Ta", "Tb"], cross=True)
```
## Example 3: Testing treatment interaction in a factorial design
```r
avg_comparisons(mod,
variables = "Ta",
by = "Tb",
hypothesis = "b2 - b1 = 0")
```
```python
avg_comparisons(mod, variables = "Ta", by = "Tb",
hypothesis = "b1 - b0 = 0")
```
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `avg_comparisons`, `comparisons`, `avg_predictions`, `predictions`, `avg_slopes`, `slopes`, `plot_predictions`, `plot_slopes`, `datagrid`, `glm`, `lm` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `avg_comparisons`, `comparisons`, `avg_predictions`, `predictions`, `avg_slopes`, `slopes`, `plot_predictions`, `plot_slopes`, `datagrid`, `logit`, `ols` for this chapter.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
# Interactions and polynomials {#sec-interactions}
- This chapter applies the workflow, framework, and software from earlier parts of the book to interpret estimates from more complex model specifications.
- The two main goals are addressing heterogeneity (when the association between $X$ and $Y$ varies by context or group) and increasing model flexibility (capturing non-linear relationships).
- Heterogeneity, moderation, interaction, effect modification, and context-conditionality are used interchangeably to describe situations where the strength of association between two variables depends on a third variable (the moderator).
- Two modeling strategies are covered: multiplicative interactions and polynomial regression.
- Even for complex models, the same interpretive quantities -- predictions, counterfactual comparisons, and slopes -- remain the primary tools for understanding results.
- Raw coefficient estimates from models with interactions or polynomials are typically difficult to interpret directly; the `marginaleffects` package provides functions to extract meaningful quantities instead.
- The ideas and workflows in this chapter extend naturally to even more flexible models (e.g., GAMs, splines, machine learning), covered in later chapters.
## Multiplicative interactions {#sec-interactions_multiplicative}
- Heterogeneity means the strength of association between an explanator $X$ and an outcome $Y$ varies based on the value of a moderator $M$; the association can be stronger, weaker, or completely reversed for different values of $M$.
- Multiplicative interactions are created by multiplying the explanator $X$ by the moderator $M$ and inserting the product as a predictor alongside the individual components.
- A standard linear interaction model is: $Y = \beta_1 + \beta_2 X + \beta_3 M + \beta_4 X \cdot M + \varepsilon$.
- In this model, when $M=0$ the effect of moving from $X=0$ to $X=1$ is $\beta_2$; when $M=1$ the effect is $\beta_2 + \beta_4$. The difference illustrates moderation.
- In R, interactions are specified with `:` (e.g., `Y ~ X + M + X:M`) or the `*` shortcut (e.g., `Y ~ X * M`), which includes both main effects and their interaction.
- In Python (statsmodels), the same `*` syntax works in model formulas: `logit("Y ~ X * M", data=df).fit()`.
- Once models become non-linear or include multiple interactions, direct interpretation of coefficients becomes impractical; the focus should be on predictions, comparisons, and slopes.
- The presentation strategy depends on whether $X$ and $M$ are categorical or continuous, leading to four sub-cases below.
### Categorical-by-categorical
- This sub-case addresses when both the explanator $X$ and the moderator $M$ are categorical (e.g., a binary treatment moderated by a three-level grouping variable).
- **Marginal predictions:** Use `avg_predictions(mod, by = c("X", "M"))` to compute the average predicted outcome for each combination of $X$ and $M$. This is equivalent to computing fitted values for every row and averaging by subgroup.
- **Visualizing predictions:** Use `plot_predictions(mod, by = c("M", "X"))` to see how predicted probabilities vary across subgroups. For instance, triangles (representing $X=1$) systematically above circles ($X=0$) indicate a positive effect of $X$.
- **Does $X$ affect $Y$?** Use `avg_comparisons(mod, variables = "X")` to compute the average counterfactual effect of $X$ on $Y$. This modifies the dataset to fix $X=0$ and $X=1$ for all observations and averages the prediction differences.
- **Is the effect moderated by $M$?** Add the `by` argument: `avg_comparisons(mod, variables = "X", by = "M")` to see whether the effect of $X$ differs across levels of $M$.
- **Hypothesis testing for moderation:** Use the `hypothesis` argument to formally test whether effects differ across moderator levels, e.g., `avg_comparisons(mod, variables = "X", by = "M", hypothesis = "b3 - b1 = 0")` in R (0-indexed `"b2 - b0 = 0"` in Python).
- A statistically significant difference (small $p$ value, large $z$ statistic) means we can reject the null hypothesis that $X$ has the same effect across sub-populations defined by $M$.
### Categorical-by-continuous
- This sub-case addresses when the explanator $X$ is categorical and the moderator $M$ is continuous.
- **Conditional predictions:** When $M$ is continuous, reporting averages for every combination of $X$ and $M$ is impractical. Instead, use `predictions(mod, newdata = datagrid(X = c(0, 1), M = fivenum))` to evaluate predictions at Tukey's five-number summary of $M$ (minimum, lower-hinge, median, upper-hinge, maximum).
- **Visualizing predictions:** `plot_predictions(mod, condition = c("M", "X"))` places the continuous moderator on the x-axis with separate lines for each value of $X$, revealing how the relationship changes over the range of $M$.
- **Does $X$ affect $Y$?** Use `avg_comparisons(mod, variables = "X")` for the overall average effect.
- **Is the effect moderated by $M$?** Use `comparisons(mod, variables = "X", newdata = datagrid(M = range))` to estimate the effect of $X$ at the minimum and maximum of $M$, then test their equality with `hypothesis = "b2 - b1 = 0"`.
- If the difference is statistically significant, we reject the null that $M$ has no moderating effect on the relationship between $X$ and $Y$.
### Continuous-by-continuous
- This sub-case addresses when both $X$ and $M$ are continuous numeric variables.
- **Conditional predictions:** Use `predictions(mod, newdata = datagrid(X = c(-2, 2), M = c(-1, 0, 1)))` to evaluate predicted outcomes at meaningful combinations of $X$ and $M$.
- **Visualizing predictions:** `plot_predictions(mod, condition = c("X", "M"))` plots predicted outcomes with the focal variable on the x-axis and lines for different moderator values, revealing how the relationship can flip direction depending on $M$.
- **Does $X$ affect $Y$?** Since both variables are continuous, the `slopes()` function is natural: `avg_slopes(mod, variables = "X")` gives the average partial derivative of $Y$ with respect to $X$ across all observed values of $M$.
- **Slopes at specific moderator values:** `slopes(mod, variables = "X", newdata = datagrid(M = fivenum))` evaluates the slope at five summary values of $M$, showing where the slope is positive, negative, or near zero.
- **Visualizing slopes:** `plot_slopes(mod, variables = "X", condition = "M")` shows how the slope of $Y$ with respect to $X$ varies continuously over the range of $M$. Adding `geom_hline(yintercept = 0)` highlights where the effect changes sign.
- **Hypothesis testing for moderation:** Compare slopes at the minimum and maximum of $M$ using `slopes(mod, variables = "X", newdata = datagrid(M = range), hypothesis = "b2 - b1 = 0")`. A significant result confirms that $M$ moderates the $X$-$Y$ relationship.
### Multiple interactions
- When more than two variables are included in multiplicative interactions (e.g., `Y ~ X * M1 * M2`), the model captures complex patterns of moderation but has notable downsides: risk of overfitting and substantially increased sample size requirements.
- Despite the complexity, the same `marginaleffects` functions and workflows apply directly.
- **Marginal predictions:** `plot_predictions(mod, by = c("X", "M1", "M2"))` visualizes predicted outcomes across all predictor combinations; facets, point shapes, or colors represent different moderator values.
- **Does $X$ affect $Y$?** `avg_comparisons(mod, variables = "X")` gives the overall average effect.
- **First-order moderation (does $M_1$ moderate the effect of $X$?):** Use `avg_comparisons(mod, variables = "X", by = "M1")` and test with `hypothesis = "b2 - b1 = 0"`.
- **Second-order moderation (does $M_2$ moderate the moderation effect of $M_1$?):** Compute comparisons by all moderator combinations with `avg_comparisons(mod, variables = "X", by = c("M2", "M1"))`, then use a difference-in-differences hypothesis: `hypothesis = "(b2 - b1) - (b4 - b3) = 0"`.
- Second-order moderation tests may lack power; even if the point estimate suggests a moderating effect, the $p$ value may not cross conventional significance thresholds.
## Polynomial regression {#sec-interactions_polynomials}
- Polynomial regression models the relationship between $Y$ and $X$ as an nth-degree polynomial: $Y = \beta_0 + \beta_1 X + \beta_2 X^2 + \cdots + \beta_n X^n + \varepsilon$.
- The model is linear in the coefficients but polynomial in $X$, making it a special case of multiplicative interactions where predictors are interacted with themselves.
- Polynomial terms are specified directly in the model formula: `lm(Y ~ X + I(X^2) + I(X^3), data = dat)` in R and `ols("Y ~ X + I(X**2) + I(X**3)", data=df).fit()` in Python. It is important to use this approach rather than creating new variables in the dataset before fitting.
- **Advantages:** Flexible; can capture curvilinear relationships that linear models miss; the degree of the polynomial is easily adjustable.
- **Disadvantages:** Risk of overfitting at high degrees; unreliable extrapolation outside the observed range.
- **Interpreting slopes:** Because the slope of $Y$ with respect to $X$ changes across the range of $X$, use `slopes(mod_cubic, variables = "X", newdata = datagrid(X = c(-2, 0, 2)))` to evaluate the marginal effect at specific points.
- **Polynomial interactions with a moderator:** Interact the moderator with all polynomial terms using parentheses as a shortcut: `lm(Y ~ M * (X + I(X^2) + I(X^3)), data = dat)`. This captures curvilinear relationships that differ across levels of $M$.
- The same `plot_predictions()`, `slopes()`, and `plot_slopes()` functions apply to polynomial models, allowing visualization and formal testing of how the $X$-$Y$ relationship varies across values of $X$ and $M$.
# Examples
- The examples below are drawn from the original chapter and illustrate representative workflows.
## Example 1: Testing moderation in a categorical-by-categorical interaction
```r
# Fit logistic regression with interaction
mod = glm(Y ~ X * M, data = dat, family = binomial)
# Average predictions by subgroup
avg_predictions(mod, by = c("X", "M"))
# Average effect of X, moderated by M
avg_comparisons(mod, variables = "X", by = "M")
# Test whether effect differs between M categories
avg_comparisons(mod, variables = "X", by = "M",
hypothesis = "b3 - b1 = 0")
```
## Example 2: Slopes with hypothesis testing in a continuous-by-continuous interaction
```r
# Compare slopes at min and max of moderator M
slopes(mod, variables = "X",
newdata = datagrid(M = range),
hypothesis = "b2 - b1 = 0")
```
## Example 3: Polynomial regression with interaction and slope evaluation
```r
# Fit cubic polynomial interacted with moderator M
mod_cubic_interaction = lm(
Y ~ M * (X + I(X^2) + I(X^3)), data = dat)
# Evaluate slopes at five-number summary of X, for each M
slopes(mod_cubic_interaction, variables = "X",
newdata = datagrid(M = c(0, 1), X = fivenum))
```
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `predictions`, `avg_predictions`, `plot_predictions`, `comparisons`, `avg_comparisons`, `datagrid`, `polr` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `predictions`, `avg_predictions`, `comparisons`, `avg_comparisons`, `plot_predictions`, `datagrid` for this chapter.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
# Categorical and ordinal outcomes {#sec-categorical}
- This chapter shows how the framework and tools introduced in Parts I and II help us give meaning to estimates obtained by fitting a categorical or ordinal outcome model.
- A categorical outcome variable is discrete and involves a finite number of categories (e.g., choice of transportation: car, bus, bike, walk). A popular approach is to fit a multinomial logit model.
- An ordinal outcome variable is discrete, involves a finite number of categories, and has a natural ordering (e.g., levels of satisfaction). A popular approach is to fit an ordered probit model.
- In an ordered probit model, the probability of each outcome category $j$ is modeled as: $$P(Y = j) = F(\theta_j - \mathbf{X}\beta) - F(\theta_{j-1} - \mathbf{X}\beta)$$ where $\mathbf{X}$ is the vector of predictors, $\beta$ is the vector of coefficients, $\theta_j$ and $\theta_{j-1}$ are threshold parameters defining boundaries between categories, and $F$ is the standard normal CDF.
- Models like the multinomial logit or ordered probit estimate different parameters for each level of the outcome, allowing predictions, comparisons, and slopes to be computed for each outcome level.
- The chapter uses Fair's (1978) dataset of 601 survey responses on extramarital affairs, with demographic and marital predictors. The outcome variable records the self-reported frequency of affairs: 0, 1, 2, 3, 4--10, or >10.
- An ordered probit model is fitted using `polr()` from the `MASS` package, with predictors: `children`, `yearsmarried`, and `gender`.
- The raw coefficients of the ordered probit measure changes in a latent variable and lack a straightforward interpretation. Instead, we transform them into predictions, counterfactual comparisons, and slopes.
## Predictions
- The first step is to compute average predictions. In a categorical outcome model, predictions are expressed as probabilities, with one prediction for every level of the outcome variable.
- To predict outcomes for a specific profile (e.g., a woman with children married for 10 years), use `datagrid()` to build a grid and `predictions()` to compute predicted probabilities for each outcome level.
- Instead of individual-level predictions, `avg_predictions()` computes average predicted probabilities across all individuals in the dataset (marginal predictions).
- The `avg_predictions()` function automatically returns one average predicted probability per outcome level. The group identifiers are in the `group` column of the output data frame.
- Subgroup predictions can be computed using the `by` argument (e.g., `avg_predictions(mod, by = "children")`) to compare average predicted probabilities for respondents with and without children.
- The `plot_predictions()` function visualizes average predicted probabilities by outcome level and subgroup, using the `by` argument to marginalize across combinations.
- Outcome categories can be combined using a custom function passed to the `hypothesis` argument of `avg_predictions()`. For example, one can collapse all non-zero affair categories to estimate the probability of reporting at least one affair.
- The custom `hypothesis` function accepts the same data frame returned by the base call (with `group`, `term`, and `estimate` columns) and returns a new data frame with `term` and `estimate` columns.
## Counterfactual comparisons
- After predictions, the chapter adopts a counterfactual perspective to quantify the strength of association between predictors and the outcome.
- The `avg_comparisons()` function estimates what would happen to the reported number of affairs if a predictor changed while holding others constant. The shift in the focal predictor is defined using the `variables` argument.
- For example, `avg_comparisons(mod, variables = list(yearsmarried = 5))` estimates the average effect of a 5-unit increase in years married on the predicted probability of each outcome level.
- Increasing `yearsmarried` by 5 reduces the predicted probability that `affairs` equals zero and increases the predicted probability of higher affair counts; these effects are statistically significant.
- In contrast, the counterfactual effect of `gender` yields small and statistically insignificant differences in predicted probabilities: we cannot reject the null that `gender` has no effect on `affairs`.
- For each outcome level, `avg_comparisons()` returns a separate row of estimates, allowing detailed inspection of how a predictor affects the entire distribution of the categorical outcome.
- The chapter concludes that transforming ordered probit parameters into predictions and comparisons yields interpretable, intuitive quantities that facilitate clear communication of findings.
# Examples
- The examples below are drawn from the original chapter and illustrate representative workflows.
## Example 1: Predicted probabilities for a specific profile
```{r}
#| cache: false
#| echo: false
#| warning: false
source("code/load.R")
```
```{r}
library(MASS)
library(marginaleffects)
dat = get_dataset("affairs")
mod = polr(
affairs ~ children + yearsmarried + gender,
method = "probit", data = dat, Hess = TRUE)
predictions(mod, newdata = datagrid(
children = "yes",
yearsmarried = 10,
gender = "woman"))
```
## Example 2: Average predictions by subgroup
```{r}
avg_predictions(mod, by = "children")
```
## Example 3: Counterfactual comparisons for years married
```{r}
avg_comparisons(mod, variables = list(yearsmarried = 5))
```
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `brm`, `glmmTMB`, `avg_comparisons`, `avg_predictions`, `predictions`, `comparisons`, `prior`, `get_draws`, `datagrid`, `ggplot` for this chapter.
- Python: at the time of writing, the features illustrated in this chapter were not yet supported in Python. Check the `marginaleffects` Python documentation for updates.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
# Multilevel regression with poststratification {#sec-mrp}
- This chapter shows how `marginaleffects` can help analysts make sense of complex hierarchical data and draw inference from unrepresentative samples, using an empirical strategy called multilevel regression with poststratification (MRP).
- The chapter has three main objectives: (1) using `marginaleffects` to interpret estimates from multilevel regression models, (2) demonstrating a consistent post-estimation workflow for both frequentist and Bayesian models, and (3) illustrating poststratification to account for unrepresentative sampling.
- MRP is applied to survey data from the 2020 Cooperative Election Study (CES) in the United States, following @Orn2023.
- The substantive goal is to estimate the level of support, in each American state, for budget cuts to police forces.
- The outcome variable is `defund`, a binary indicator equal to 1 if a respondent supports cuts to police budgets.
- Predictors include individual-level covariates: `gender`, `race`, `age`, `education`, `military` experience, and the respondent's `state`.
- The `ces_survey` dataset includes 3000 observations, randomly drawn from the full CES data.
- This case study only includes code for R; the Python features illustrated here were not yet supported at the time of writing.
## Multilevel models
- Multilevel (or mixed effects) models are a popular strategy to study data that have a hierarchical, nested, or multilevel structure.
- Examples of nested data include survey respondents in different states; repeated measures made on the same subjects or on clustered observations; and students nested in classrooms, schools, districts, and states.
- The parameters of a multilevel model can be divided into two types: "fixed effects" (assumed constant across all groups) and "random effects" (allowed to vary across groups).
- A major benefit of mixed effects models is that they allow variation in parameters between subsets of the data. For example, a "random intercept" lets the baseline level of support for `defund` vary across US states. A "random coefficient" for `gender` allows the association between `gender` and `defund` to vary from state to state.
- Importantly, random parameters are not completely free to vary from group to group. The model imposes constraints on the distribution of parameters, which regularizes estimates. This partial pooling allows estimates for groups with small sample sizes to borrow strength from groups with larger sample sizes, reducing overfitting and stabilizing estimates.
- A thorough introduction to multilevel modeling lies outside the scope of this chapter, but interested readers can refer to texts such as @GelHil2006, @FinHolBolKel2019, @Hod2021, and @Bur2024.
## Frequentist
- Several software packages in R allow us to fit mixed effects models from a frequentist perspective, such as `lme4` and `glmmTMB`.
- Models are defined using the familiar formula syntax. Fixed components are specified like any other predictor; random effects use a special syntax with parentheses and a vertical bar, e.g., `y ~ x + z + (1 + z | group)`.
- In this formula, `1` is a random intercept allowing the baseline of `y` to vary across groups, and `z` is a random coefficient allowing the association between `z` and `y` to vary across groups.
- The chapter fits a logistic regression model with `glmmTMB`: `defund ~ age + education + military + gender + (1 + gender | state)`, including a random intercept and random `gender` coefficient by state.
- The `predictions()` function is used to compute predicted probabilities for hypothetical individuals (e.g., one from California and one from Alabama), illustrating how random intercepts produce different baseline probabilities across states.
- The `avg_comparisons()` function is used to assess the strength of association between `age` and `defund`. Older respondents (70+) are substantially less likely to support police budget cuts than younger respondents (18-29).
- Uncertainty estimates for frequentist mixed effects models only account for variability in fixed effects parameters, not random effects parameters. Bayesian modeling allows more options for uncertainty quantification.
- The same `marginaleffects` workflow from earlier chapters applies directly to mixed effects models.
## Bayesian
- Bayesian regression analysis has a long history in statistics. Recent developments in computing, algorithms, and software have dramatically lowered the barriers to entry for Bayesian mixed effects models.
- A typical Bayesian analysis involves several iterative steps: model formulation, prior specification, model refinement, estimation, and interpretation, forming a "Bayesian workflow" (@Gel2020).
- This section uses the `brms` package for R to fit Bayesian mixed effects models, and shows how `marginaleffects` facilitates two important steps: prior predictive checks and posterior summaries.
- The model has the same structure as the frequentist model: `defund ~ age + education + military + gender + (1 + gender | state)`.
- The `age`, `education`, and `military` variables are associated with fixed effect parameters, while the `gender` and intercept parameters are allowed to vary from state to state.
- Interested readers can refer to @Gel2013, @McE2020, or @Bur2024 for comprehensive treatments of Bayesian modeling.
### Prior predictive checks
- One key difference between Bayesian and frequentist analysis is that Bayesian analysts must explicitly specify priors over model parameters.
- Priors encode the knowledge, beliefs, or information that the analyst held about the parameters before looking at the data. Some priors can have an important impact on results, so they must be chosen carefully.
- Vague or diffuse priors (e.g., `normal(0, 1e6)`) express high uncertainty about parameter values. Informative or narrow priors (e.g., `normal(0, 0.2)`) express confidence that parameters are closer to zero.
- Choosing priors is difficult partly because priors are specified on the scale of model parameters (e.g., log-odds coefficients), which can be unintuitive.
- Prior predictive checks address this difficulty: by setting `sample_prior = "only"` in `brm()`, we can simulate quantities of interest from the model and priors without using any data. This lets analysts evaluate whether priors produce sensible predictions before being influenced by observed data.
- With `marginaleffects`, we can post-process prior-only models to compute predictions, comparisons, or slopes on more intuitive scales (e.g., predicted probabilities rather than log-odds coefficients).
- When using vague priors, average predicted probabilities are centered at 0.5 with credible intervals covering nearly the full unit interval. Informative priors produce much narrower intervals.
- We can also conduct prior predictive checks directly on treatment effects (e.g., `avg_comparisons(model_vague, variables = "military")`), allowing analysts to verify that implied prior effect sizes are reasonable.
### Posterior summaries
- After fitting the model to data (by dropping `sample_prior`), we use `marginaleffects` to summarize the posterior distribution of quantities of interest.
- By default, `marginaleffects` functions report the mean of posterior draws along with equal-tailed credible intervals.
- `avg_predictions(model, by = "military")` computes the average predicted probability of supporting `defund` for respondents with and without military experience. People without military experience have a higher estimated probability of supporting cuts.
- `avg_comparisons(model, variables = list(age = c("18-29", "70+")))` measures the association between age and `defund`. Moving from the youngest to the oldest age category is associated with a substantial decrease in the probability of supporting funding cuts.
- Since the model allows the `gender` parameter to vary by state, `comparisons(model, variables = "gender", newdata = datagrid(state = unique))` computes state-specific risk differences, which can be plotted to show how the gender-defund association varies geographically. The estimated association is quite stable across states.
- The `get_draws()` function extracts draws from the posterior distribution, returning data frames (wide or long), matrices, or `rvar` distribution objects compatible with the `posterior` package.
- With posterior draws in hand, analysts can compute custom posterior summaries using standard R functions, such as the proportion of the posterior density above a given threshold.
## Poststratification {#sec-mrp_poststratification}
- Even with a large sample (3000 observations), some states are much more populous than others, and the dataset includes very few observations from certain areas, making direct state-level estimates unreliable.
- This kind of sampling disparity occurs in many contexts: estimating voting intentions from national surveys, measuring well-being from web surveys that oversample certain demographics, or estimating vaccination rates from data skewed toward affluent neighborhoods.
- MRP addresses this by combining a multilevel model with poststratification in four steps: (1) estimate a mixed-effects model, (2) construct a poststratification frame, (3) predict for each cell of the frame, and (4) compute a weighted average.
- The poststratification frame (`ces_demographics`) records the prevalence of each socio-demographic profile within each state, typically sourced from the census. The proportions must sum to one for each state.
- To make predictions for every row of the poststratification frame, use `predictions(model, newdata = demographics)`.
- To compute poststratified state-level estimates, use `avg_predictions(model, newdata = demographics, wts = "percent", by = "state")`. This produces one estimate per state, adjusted for the demographic composition.
- Posterior draws can be extracted with `get_draws()` and visualized using the `ggdist` package to show the full posterior density for each state, revealing that support for police budget cuts varies across states (e.g., higher in California, lower in Wyoming).
- The accuracy of MRP estimates depends on the predictive performance of the first-stage model and may be affected by standard problems such as overfitting.
# Examples
- The examples below are drawn from the original chapter and illustrate representative workflows.
## Example 1: Frequentist mixed effects model
```r
library(glmmTMB)
library(marginaleffects)
survey = get_dataset("ces_survey")
mod = glmmTMB(
defund ~ age + education + military + gender + (1 + gender | state),
family = binomial,
data = survey)
# Predicted probability for hypothetical individuals in CA vs AL
predictions(mod, newdata = datagrid(
state = c("CA", "AL"),
gender = "Man",
military = 0,
education = "4 year",
age = "50-59"
))
# Average comparison for age
avg_comparisons(mod, variables = "age")
```
## Example 2: Bayesian prior predictive check
```r
library(brms)
library(marginaleffects)
priors_informative = c(
prior(normal(0, 0.2), class = "b"),
prior(normal(0, 0.2), class = "Intercept")
)
model_informative = brm(
defund ~ age + education + military + gender + (1 + gender | state),
family = bernoulli,
prior = priors_informative,
sample_prior = "only",
data = survey)
# Prior predictive check: average predictions by gender
avg_predictions(model_informative, by = "gender")
# Prior predictive check on a treatment effect
avg_comparisons(model_informative, variables = "military")
```
## Example 3: Poststratification
```r
library(brms)
library(marginaleffects)
# Assume 'model' is a fitted brms model
demographics = get_dataset("ces_demographics")
# State-level estimates via MRP
p = avg_predictions(model,
newdata = demographics,
wts = "percent",
by = "state")
head(p)
# Extract posterior draws and plot
library(ggdist)
library(ggplot2)
p = p |>
get_draws(shape = "rvar") |>
sort_by(~ estimate) |>
transform(state = factor(state, levels = state))
ggplot(p, aes(y = state, xdist = rvar)) +
stat_slab(height = 2, color = "white") +
labs(x = "Posterior density", y = NULL) +
xlim(c(.3, .5))
```
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `predictions`, `avg_predictions`, `plot_predictions`, `avg_comparisons`, `comparisons`, `datagrid` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `predictions`, `comparisons`, `slopes`, `avg_predictions`, `avg_comparisons`, `avg_slopes` for this chapter.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
# Machine learning {#sec-ml}
- The concepts and post-estimation tools introduced in earlier chapters---predictions, counterfactual comparisons, and slopes---are largely model-agnostic; they are applicable to both statistical and machine learning approaches.
- These tools are especially effective for model description, a task that is very important in machine learning applications, where analysts need to audit and understand how models respond to different inputs.
- Auditing and describing machine learning models is essential to ensure that predictions remain fair, and that they are driven by factors compatible with the substantive knowledge of domain experts.
- For instance, in credit scoring systems, evaluating how variations in applicant characteristics---such as income, employment status, or ethnicity---influence creditworthiness assessments helps detect and mitigate potential biases.
- Similarly, in hiring algorithms, how models weight different candidate attributes---like education level or years of experience---can help recruiters use models in decision-making.
- Audits and model description are crucial to improve the transparency and interpretability of data analyses.
- The `marginaleffects` package facilitates model description and auditing by allowing analysts to compute and visualize predictions, counterfactual comparisons, and slopes.
- It integrates seamlessly with some of the most prominent machine learning frameworks in R (`tidymodels` and `mlr3`) and Python (Scikit Learn).
## `tidymodels` and `mlr3`
- `tidymodels` is a collection of packages in R designed for modeling and machine learning using `tidyverse` principles, offering a cohesive interface for data preprocessing, modeling, and validation.
- `mlr3` is a modern, object-oriented framework in R that provides a comprehensive suite of tools for machine learning, including a wide array of algorithms, resampling methods, and performance measures.
- By supporting both `tidymodels` and `mlr3`, `marginaleffects` enables users to interpret a wide variety of machine learning models.
- Scikit Learn is a powerful Python library for machine learning that provides simple and efficient tools for data mining and data analysis.
- A comprehensive introduction to machine learning in general, or to particular frameworks, lies outside the scope of this book. Instead, the chapter shows a very simple example to demonstrate that the workflow built up in previous chapters applies in straightforward fashion to this new context.
- The chapter uses data on Airbnb rental properties in London, which includes information on over 50,000 units with features such as unit type, number of bedrooms, parking, and internet access; the primary outcome is the rental price.
- The data is split into a training set for fitting the model and a test set for evaluating predictions.
- The `boost_tree()` function specifies the model type (XGBoost boosted trees for regression), `recipe()` identifies the outcome variable and initiates data preprocessing, `step_dummy()` converts categorical predictors into dummy variables, and `workflow()` combines the model and recipe before calling `fit()`.
- Users who prefer a different algorithm can swap `boost_tree()` for `linear_reg()`, `rand_forest()`, `bart()`, etc.
## Predictions {#sec-ml_predictions}
- With the fitted model in hand, the `predictions()` function generates predictions in the test set.
- As usual, `predictions()` returns a simple data frame with the quantity of interest in the `estimate` column, and the original data in separate columns.
- The quality of predictions can be checked by plotting predicted values (`estimate`) against observed values (`price`); points on the diagonal are correctly predicted.
- There is considerable spread around the diagonal, which means the algorithm makes substantial prediction errors.
- Most of the standard functions and arguments in `marginaleffects` are available for ML models.
- For instance, `avg_predictions()` with the `by` argument can compute the average predicted price by unit type (e.g., private rooms vs. entire homes).
- When `marginaleffects` functions are applied to models fitted by `tidymodels` or `mlr3`, standard errors are not available because ML models are not typically accompanied by a variance-covariance matrix; however, `tidymodels` has built-in support for some uncertainty quantification strategies like conformal prediction.
### Partial dependence plot
- A Partial Dependence Plot (PDP) is a strategy to visualize how predictions change with certain predictors, by computing predictions over a range of values for a predictor and averaging over other variables.
- This is useful for understanding complex models.
- The `plot_predictions()` function in `marginaleffects` simplifies the creation of these plots by computing average predicted outcomes for each combination of focal variables and plotting the results.
- In the Airbnb example, the PDP shows that the price of a single private room does not really change as the total number of bedrooms increases, while the price of renting an entire unit does increase with the number of bedrooms.
- In some contexts, analysts prefer partial dependence plots based on a counterfactual grid, which duplicates the dataset for every combination of focal variable values, ensuring the distribution of other covariates is identical across combinations.
- Counterfactual-grid PDPs can be interpreted as illustrating "all else equal" predictions; they are constructed using `datagrid()` with `grid_type = "counterfactual"` and then passed to `plot_predictions()`.
- When the dataset is very large, a random subset can be used to build the counterfactual grid to manage memory usage.
## Counterfactual comparisons {#sec-ml_comparisons}
- The `avg_comparisons()` function answers counterfactual queries such as: on average, how does the predicted price change when we increase the number of bedrooms by 2, holding all other variables constant?
- The `variables` argument specifies which predictors to change and by how much (e.g., `list(bedrooms = 2)` increases bedrooms by 2).
- In the Airbnb example, the model predicts that a unit with two extra bedrooms will command a meaningfully higher price.
- To examine the combined effect of changing multiple predictors simultaneously---such as adding one bedroom and transitioning from no wireless internet to having it---the `cross` argument is used with `avg_comparisons()`.
- In the Airbnb example, adding one bedroom and wireless internet access to a rental unit increases the expected price.
- The integration of machine learning models with tools like `marginaleffects` allows for a deeper understanding and interpretation of complex models.
- By leveraging predictions, counterfactual comparisons, and partial dependence plots, analysts can gain insights into model behavior and ensure that predictions align with domain knowledge.
- This approach enhances model transparency and aids in making informed decisions based on model outputs.
# Examples
- The examples below are drawn from the original chapter and illustrate representative workflows.
## Example 1: Fit an XGBoost model and generate predictions
```r
library(tidymodels)
library(marginaleffects)
set.seed(48103)
airbnb <- get_dataset("airbnb")
airbnb_split <- initial_split(airbnb)
train <- training(airbnb_split)
test <- testing(airbnb_split)
xgb <- boost_tree(mode = "regression", engine = "xgboost")
mod <- recipe(airbnb, price ~ .) |>
step_dummy(all_nominal_predictors()) |>
workflow(spec = xgb) |>
fit(train)
predictions(mod, newdata = test)
```
## Example 2: Partial dependence plot
```r
plot_predictions(mod,
by = c("bedrooms", "unit_type"),
newdata = airbnb) +
labs(x = "# Bedrooms", y = "Predicted Price", linetype = "")
```
## Example 3: Counterfactual comparisons with the cross argument
```r
avg_comparisons(mod,
variables = list(bedrooms = 2),
newdata = airbnb)
avg_comparisons(mod,
variables = c("bedrooms", "Wireless Internet"),
cross = TRUE,
newdata = airbnb)
```
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `hypotheses`, `predictions`, `comparisons`, `slopes`, `avg_comparisons`, `avg_predictions`, `avg_slopes`, `inferences`, `vcov`, `vcovHC` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `predictions`, `comparisons`, `slopes`, `avg_predictions`, `avg_comparisons`, `avg_slopes`, `hypotheses` for this chapter.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
# Uncertainty {#sec-uncertainty}
- This chapter introduces four approaches to quantify uncertainty around quantities of interest: the delta method, bootstrap, simulation-based inference, and conformal prediction.
- The delta method approximates the variance of a function of random variables using a linear (Taylor series) approximation; it is fast but relies on asymptotic normality.
- The bootstrap resamples the observed data with replacement, refits the model many times, and uses the resulting distribution to build confidence intervals.
- Simulation-based inference draws parameter sets from a multivariate normal distribution centered on the estimated coefficients and covariance matrix, then computes the quantity of interest for each draw.
- Conformal prediction constructs prediction intervals that are valid in finite samples, without distributional assumptions, and even under model misspecification, provided the data are exchangeable.
- The `marginaleffects` package provides convenient implementations of all four approaches through its core functions and the `inferences()` wrapper.
- The delta method is the default uncertainty quantification strategy in `marginaleffects`.
- Readers unfamiliar with the basics of multivariable calculus may wish to skip this chapter and rely on the package defaults.
## Delta method {#sec-uncertainty_delta_method}
- The delta method is a statistical technique used to approximate the variance of a function of random variables.
- Data analysts often want to compare regression coefficients or predicted probabilities by subtracting them from one another; such comparisons are functions of model parameters whose variances are hard to derive analytically.
- The delta method provides a convenient way to approximate those variances, enabling uncertainty quantification around most quantities of interest in this book.
- The technique is useful, versatile, and fast, but relies on a coarse linear approximation and the asymptotic normality of the estimator.
- It is only valid when the function of parameters is continuously differentiable in a neighborhood of its parameters.
- When analysts do not believe the delta method's conditions hold, they can turn to the bootstrap or simulation-based inference as alternatives.
- The theoretical properties are discussed and proved in many textbooks; this chapter focuses on computation rather than proofs.
- The rest of this section provides hands-on tutorials for the univariate case, the multivariate case, and the use of robust or clustered standard errors.
### Univariate delta method
- The univariate delta method computes the standard error of a function of one parameter, illustrated here with the natural logarithm of a single regression coefficient.
- Consider a linear regression $Y = \beta_1 + \beta_2 X + \beta_3 Z + \varepsilon$; the goal is to find $\text{Var}\left[\log(\hat{\beta}_2)\right]$.
- A first-order Taylor series expansion approximates $\log(\hat{\beta}_2) \approx \log(\beta_2) + \frac{1}{\beta_2}(\hat{\beta}_2 - \beta_2)$.
- Inserting this approximation into the variance operator and using the scaling property of variances yields $\text{Var}\left[\log(\hat{\beta}_2)\right] \approx \frac{1}{\beta_2^2}\text{Var}\left[\hat{\beta}_2\right]$.
- Since the true $\beta_2$ is unknown, we use the plug-in estimate $\hat{\beta}_2$ in practice.
- The chapter demonstrates this formula by simulating data, fitting a linear model, and computing the standard error of $\log(\hat{\beta}_2)$ manually.
- The result is verified against the `hypotheses()` function from `marginaleffects`, which computes arbitrary functions of model parameters along with delta method standard errors.
- This shows that `hypotheses(mod, "log(b2) = 0")` produces the same estimate and standard error as the manual calculation.
### Multivariate delta method
- The univariate delta method generalizes to the multivariate case with a vector of input parameters $\mathcal{B}=\{\beta_1,\beta_2,\ldots,\beta_k\}$ and a vector-valued output $\Theta=\{\theta_1,\theta_2,\ldots,\theta_n\}$.
- The multivariate delta method formula is $\operatorname{Var}\left(h(\mathcal{B})\right) = J^T \cdot \operatorname{Var}\left(\mathcal{B}\right) \cdot J$, where $J$ is the Jacobian matrix of partial derivatives.
- The Jacobian $J$ has rows equal to the number of input parameters and columns equal to the number of output quantities; element $(i,j)$ is $\partial \theta_j / \partial \beta_i$.
- The chapter illustrates this with $\theta = \beta_2 - \beta_3$ (the difference between two coefficients), for which the Jacobian is $J = [0, 1, -1]^T$.
- The variance-covariance matrix $\operatorname{Var}(\mathcal{B})$ is extracted using the `vcov()` function in R.
- The standard error is computed as $\sqrt{J^T \cdot V \cdot J}$ using matrix multiplication.
- The manual result is verified to be identical to `hypotheses(mod, hypothesis = "b2 - b3 = 0")`.
### Robust or clustered standard errors {#sec-uncertainty_robust}
- Classical variance-covariance estimates rely on strong assumptions that rule out autocorrelation, heteroskedasticity, and clustering.
- Autocorrelation arises when prediction errors at time $t$ are correlated with errors at time $t+1$; clustering occurs when errors are similar within groups (e.g., classrooms, neighborhoods); heteroskedasticity means the variance of errors differs across observations.
- Where such patterns occur, classical standard errors may inaccurately characterize uncertainty.
- The classical OLS variance uses a "sandwich" formula: $\text{Var}(\hat{\beta}) = (X^TX)^{-1}(\sigma^2_\varepsilon X^TX)(X^TX)^{-1}$, where $\sigma^2_\varepsilon$ is a single constant.
- Robust ("heteroskedasticity-consistent" or "Huber-White") standard errors replace the meat of the sandwich with a matrix using squared residuals on the diagonal, allowing the variance to differ across observations.
- The `sandwich` package in R provides `vcovHC()` and related functions that support many models and estimators.
- In `marginaleffects`, the `vcov` argument in all core functions allows users to specify robust or clustered standard errors (e.g., `vcov = "HC0"` or `vcov = ~ village`).
- Combining the delta method with robust variance-covariance estimates lets analysts account for heteroskedasticity or clustering in all quantities of interest.
## Bootstrap {#sec-framework_uncertainty_bootstrap}
- The bootstrap, pioneered by Bradley Efron in the 1970s-1980s, uses resampling to approximate the sampling distribution of a statistic.
- The algorithm has five steps: (1) sample rows with replacement, (2) fit the model to the resampled data, (3) compute the statistic, (4) repeat many times, (5) use quantiles of the bootstrap distribution to build a confidence interval.
- The chapter demonstrates this by computing a confidence interval for the difference between two regression coefficients ($\beta_3 - \beta_2$) in a linear model fit to the Thornton (2008) data.
- The `inferences()` function from `marginaleffects` provides a consistent interface to bootstrap packages (`boot`, `rsample`, `fwb`), making this workflow much easier.
- The syntax pipes any core `marginaleffects` function output into `inferences(method = "boot")` to obtain bootstrap confidence intervals.
- Bootstrapping is widely applicable even when the underlying distribution of quantities of interest is unknown.
- However, bootstrapping can be computationally intensive, especially for large datasets or complex models, because it requires refitting the model many times.
- Selecting an adequate resampling scheme can be challenging when the data-generating process is complex.
## Simulation {#sec-framework_uncertainty_simulation}
- Simulation-based inference is an alternative to the delta method and bootstrap, leveraging computational simulations to estimate variability.
- The foundational assumption is that the regression estimator is asymptotically normal, allowing draws from a multivariate normal distribution.
- The algorithm has three steps: (1) draw parameter sets from a multivariate normal with mean equal to the estimated coefficients and variance equal to the estimated covariance matrix, (2) compute the statistic for each draw, (3) build confidence intervals from quantiles of the simulated quantities.
- Early applications include Krinsky and Robb (1986); King, Tomz, and Wittenberg (2000) popularized the approach with the `clarify` software.
- As Rainey (2024) notes, this process can be thought of as an informal Bayesian posterior simulation.
- The chapter demonstrates the approach by computing a confidence interval for $\beta_3 - \beta_2$ using `rmvnorm()` from the `mvtnorm` package.
- In `marginaleffects`, the same result is obtained via `inferences(method = "simulation")`.
- Simulation-based inference is usually faster than bootstrapping (no model refitting) but more computationally expensive than the delta method.
## Conformal prediction {#sec-conformal}
- Confidence intervals from `predictions()` characterize uncertainty about the expected value of the response; they are not designed to cover a specified share of unseen data points.
- A common misunderstanding is that confidence intervals should cover future observations at their nominal rate; in fact, they typically cover far fewer out-of-sample points.
- The chapter demonstrates this with a simulation: a 90% confidence interval covers the true mean about 90% of the time, but covers out-of-sample observations at a much lower rate.
- To obtain intervals that cover a pre-specified share of new observations, analysts need prediction intervals rather than confidence intervals.
- Conformal prediction is a paradigm for creating statistically rigorous prediction intervals that are valid in finite samples, without distributional or model assumptions, provided the data are exchangeable.
- The main caveats are: (1) the algorithms require exchangeable data and do not apply to time series or spatial data with dependence; (2) coverage guarantees are marginal (averaged over random test points), not necessarily well-calibrated locally in different strata of predictors; (3) interval width depends on the quality of the prediction model and score function.
- The `inferences()` function in `marginaleffects` supports conformal prediction with methods like `"conformal_split"` and `"conformal_cv+"`.
- Two prediction tasks are illustrated: numeric outcomes (split conformal) and categorical outcomes (cross-validation conformal with softmax scores).
### Numeric outcome {#sec-uncertainty_conformal_numeric}
- The chapter uses a dataset of over 1 million US military members to predict rank from grade, branch, gender, and race using a linear model.
- Split conformal prediction divides the data into three parts: training (to fit the model), calibration (to determine interval width), and test (to evaluate predictions).
- The key assumption is that observations in the calibration set are exchangeable with those in the test set, so prediction errors should be similar across both.
- The algorithm computes absolute residuals in the calibration set, finds the 95th quantile, and constructs prediction intervals as $\hat{Y}_i \pm d$, where $d$ is the smallest absolute residual above that quantile.
- The resulting prediction intervals cover approximately 95% of out-of-sample observations in the test set, matching the target coverage rate.
- The same result is obtained more conveniently using `predictions(mod) |> inferences(method = "conformal_split", conformal_calibration = dat$calib, conformal_test = dat$test)`.
- Conformal prediction intervals are valid even if the model is misspecified, though they may not be well-calibrated locally across different regions of the predictor space.
- Prediction intervals produced by conformal methods are typically much wider than confidence intervals, because they aim to cover individual observations rather than the expected mean.
### Categorical outcome {#sec-uncertainty_conformal_categorical}
- For classification tasks, conformal prediction produces a set of possible classes for each observation, rather than a single point prediction.
- The chapter uses a multinomial logit model to predict branch of the military (air force, army, marine corps, navy) from gender and race.
- The outcome is nominal rather than numeric, so instead of residuals, prediction quality is measured using a "softmax score" derived from the probability assigned to the true class.
- Instead of a held-out calibration set, the chapter illustrates a cross-validation-based conformal algorithm (`conformal_cv+`), though split conformal would also work.
- The `inferences()` function returns a data frame with a `pred.set` column containing the list of predicted classes for each test observation.
- The chapter shows an example where the most likely point prediction ("army") is incorrect, but the conformal prediction set includes both the incorrect top prediction and the correct class ("air force").
- Checking all test observations confirms that conformal prediction sets include the true class approximately 80% of the time, matching the requested `conf_level = .8`.
- This demonstrates the value of prediction sets over point predictions for classification under uncertainty.
# Examples
- The examples below are drawn from the original chapter and illustrate representative workflows.
## Example 1: Univariate delta method standard error
```r
# Simulate data and fit a linear model
set.seed(48103)
X <- rnorm(100)
Z <- rnorm(100)
Y <- 1 * X + 0.5 * Z + rnorm(100)
dat <- data.frame(Y, X, Z)
mod <- lm(Y ~ X + Z, data = dat)
# Extract the coefficient and its variance
b2 <- coef(mod)[2]
v <- vcov(mod)[2, 2]
# Delta method standard error for log(beta_2)
se_log_b2 <- sqrt(v / b2^2)
se_log_b2
# Verify with marginaleffects
library(marginaleffects)
hypotheses(mod, "log(b2) = 0")
```
## Example 2: Bootstrap confidence interval
```r
library(marginaleffects)
dat <- get_dataset("thornton")
mod <- lm(outcome ~ agecat - 1, data = dat)
# Define the quantity of interest: difference between coefficients
statistic <- \(model) coef(model)[3] - coef(model)[2]
statistic(mod)
# Bootstrap by hand
sample_fit_compute <- function() {
index <- sample(1:nrow(dat), size = nrow(dat), replace = TRUE)
resample <- dat[index, ]
mod_resample <- lm(outcome ~ agecat - 1, data = resample)
statistic(mod_resample)
}
set.seed(48103)
boot_dist <- replicate(1000, sample_fit_compute())
quantile(boot_dist, prob = c(0.025, 0.975))
# Or use inferences() for convenience
hypotheses(mod, "b3 - b2 = 0") |>
inferences(method = "boot", R = 1000)
```
## Example 3: Conformal prediction intervals
```r
library(marginaleffects)
dat <- get_dataset("military")
# Split data into training, calibration, and test sets
set.seed(48103)
idx <- sample(c("train", "calibration", "test"),
size = nrow(dat), replace = TRUE)
dat <- split(dat, idx)
# Fit model on training data
mod <- lm(rank ~ grade + branch + gender + race, data = dat$train)
# Conformal prediction intervals via inferences()
p <- predictions(mod) |>
inferences(
method = "conformal_split",
conformal_calibration = dat$calib,
conformal_test = dat$test)
# Check coverage
covered <- p$rank > p$pred.low & p$rank < p$pred.high
mean(covered)
```
---
## Start Here: Manual Pages
- R: open function help with `?function_name` or `help("function_name")`; prioritize `predictions`, `comparisons`, `slopes`, `avg_predictions`, `avg_comparisons`, `avg_slopes`, `plot_predictions`, `plot_comparisons`, `plot_slopes`, `datagrid`, `hypotheses`, `get_dataset` for this chapter.
- Python: inspect docstrings with `help(function_name)` and package docs; prioritize `predictions`, `comparisons`, `slopes`, `avg_predictions`, `avg_comparisons`, `avg_slopes`, `plot_predictions`, `plot_comparisons`, `plot_slopes`, `datagrid`, `hypotheses`, `get_dataset`, `fit_sklearn` for this chapter.
- Before replicating examples, confirm argument defaults, return objects, and uncertainty options in both languages.
Citation: Model to Meaning: How to interpret statistical models in R and Python. Arel-Bundock, Vincent. 2026. CRC Press. routledge.com/9781032908724
# Appendix II: Python {#sec-python .unnumbered .unlisted}
- This appendix collects Python code equivalents for nearly all R commands shown throughout the book.
- The same code, with full output, is available on the marginaleffects.com website.
- By default, `marginaleffects` commands in Python return data frames in Polars format.
- Polars objects can be converted to Pandas, Numpy, or other formats using `.to_pandas()`, `.to_numpy()`, and similar `.to_*()` methods.
- Python uses 0-based indexing, whereas R uses 1-based indexing; this is important when translating examples.
- The appendix imports `polars`, `numpy`, `marginaleffects`, `plotnine`, `scipy.stats`, and `statsmodels.formula.api` as the standard working environment.
- The `get_dataset()` function retrieves built-in datasets, and optional arguments like `search` and `docs` help discover and document datasets.
- Features not yet available in the Python version of `marginaleffects` are listed in the Roadmap at the end.
## 1. Who is this book for? {.unnumbered}
### Data {.unnumbered}
- `get_dataset()` retrieves named datasets from R packages for use in Python; for example, `get_dataset("Titanic", "Stat2Data")` loads the Titanic dataset.
- Subsetting rows and columns uses Polars bracket syntax, e.g., `dat[:6, ["Name", "Survived", "Age"]]`.
- The `search` argument in `get_dataset()` lets users search for datasets by keyword.
- The `docs` argument in `get_dataset()` returns documentation about the dataset.
- These tools ensure Python users can reproduce every data example from the book.
## 3. Conceptual framework {.unnumbered}
### Predictor grids {.unnumbered}
- The `datagrid()` function creates grids of predictor values for prediction and comparison.
- An empirical grid uses the observed data directly.
- An interesting grid lets users specify particular values for selected variables, e.g., `datagrid(Bin=[0,1], newdata=dat)`.
- A representative grid sets all predictors to their mean or mode via `grid_type="mean_or_mode"`.
- A balanced grid creates all unique combinations of predictors via `grid_type="balanced"`.
- A counterfactual grid duplicates the entire dataset for each level of a focal variable via `grid_type="counterfactual"`.
- The `rowidcf` column in counterfactual grids tracks which original row each duplicated row corresponds to.
## 4. Hypothesis and equivalence tests {.unnumbered}
### Null hypothesis {.unnumbered}
- The `hypotheses()` function tests whether model parameters or functions of parameters equal a specified null value.
- Setting `hypothesis=0.5` tests the null that a coefficient equals 0.5, rather than the default null of 0.
- Linear hypothesis tests compare or combine coefficients, e.g., `hypothesis="b2 - b0 = 0"` tests whether two coefficients are equal.
- Non-linear hypothesis tests are also supported, e.g., `hypothesis="b2 / b0 = 1"` tests a ratio, and arbitrary Python expressions like `b1**2 * np.exp(b0)` can be used.
- Shortcut strings like `"difference ~ reference"` and `"ratio ~ sequential"` automate pairwise comparisons across all coefficients.
### Equivalence tests {.unnumbered}
- The `equivalence` argument defines a range (e.g., `[-0.05, 0.05]`) within which the effect is considered practically equivalent to zero.
- Equivalence tests are complementary to standard null hypothesis tests: a standard test asks whether the effect differs from zero, while an equivalence test asks whether the effect is small enough to be negligible.
- The result includes a `p_value_equiv` column for the equivalence test p-value.
- Combining a standard hypothesis test with an equivalence test helps distinguish between "no evidence of an effect" and "evidence of no meaningful effect."
- This approach directly mirrors the R workflow described in the main hypothesis chapter.
## 5. Predictions {.unnumbered}
### Quantity of interest {.unnumbered}
- Predictions are model-based expected values of the outcome, computed for specific covariate profiles.
- In logistic regression, the linear predictor is the sum of coefficient-times-predictor products; applying the logistic function converts this to a probability.
- The `predictions()` function computes predictions with standard errors and confidence intervals, given a model and a `newdata` grid.
- Predictions can be made on different scales (e.g., link vs. response) depending on the model type.
- Predictions are conditional quantities: they depend on the values of all predictors in the model.
### Predictor grids {.unnumbered}
- An empirical grid (the default) computes predictions for every row in the original data, yielding one prediction per observation.
- An interesting grid uses `datagrid()` to specify particular predictor values, e.g., `datagrid(agecat="18 to 35", incentive=[0,1])`.
- A representative grid via `newdata="mean"` creates a single synthetic profile with all-mean/all-mode values.
- A balanced grid via `newdata="balanced"` creates all combinations of categorical levels with numeric variables at their means.
- A counterfactual grid via `variables={"incentive": [0, 1]}` duplicates the dataset for each specified value of the focal variable.
- Counterfactual predictions can be visualized by plotting treatment vs. control predictions for each unit.
### Aggregation {.unnumbered}
- `avg_predictions()` computes the mean of unit-level predictions, yielding an average (marginal) predicted value.
- The `by` argument groups the average by a categorical variable, e.g., `avg_predictions(mod, by="agecat")`.
- `avg_predictions(mod, newdata="balanced", by="agecat")` averages over a balanced grid within each subgroup.
- Combining `variables` and `by` arguments computes average counterfactual predictions per subgroup.
- The result of `avg_predictions()` is equivalent to manually computing `predictions()` and taking the mean of the `estimate` column.
### Hypothesis tests {.unnumbered}
- The `hypothesis` argument in `avg_predictions()` tests whether average predictions differ across groups, e.g., `hypothesis="b2 - b1 = 0"`.
- Shortcut strings like `"difference ~ sequential"` and `"difference ~ reference"` automate pairwise comparisons.
- The pipe `|` syntax in hypothesis strings (e.g., `"difference ~ sequential | incentive"`) conducts pairwise tests within subgroups defined by another variable.
- The `equivalence` argument adds equivalence testing to average predictions, supplementing standard null hypothesis tests.
- These hypothesis and equivalence tests follow the same syntax as the `hypotheses()` function from the hypothesis chapter.
### Visualization {.unnumbered}
- Unit-level predictions can be plotted as histograms or empirical CDFs using `plotnine`, after casting variables to string types as needed.
- `plot_predictions(mod, by="incentive")` produces a marginal prediction plot, averaging across other covariates.
- Adding a second variable to `by` (e.g., `by=["incentive", "agecat"]`) creates grouped marginal plots.
- `plot_predictions(mod, condition="distance")` produces a conditional prediction plot showing how predictions vary with a continuous predictor.
- Multiple variables in the `condition` argument (up to three) create faceted or color-coded conditional plots.
- The `draw=False` argument returns the underlying data frame instead of rendering the plot, enabling custom visualizations.
## 6. Counterfactual comparisons {.unnumbered}
### First steps {.unnumbered}
- A counterfactual comparison estimates the change in predicted outcome when a focal predictor changes, holding other variables constant.
- Manually: create two grids differing only in the focal variable, compute predictions for each, and take the difference.
- The `comparisons()` function automates this, returning the difference along with standard errors and test statistics.
- Comparisons are conditional quantities: each individual may have a different comparison depending on their covariate values.
- The `newdata` argument specifies the covariate profile at which the comparison is evaluated.
### Comparison functions {.unnumbered}
- The `comparison` argument in `comparisons()` controls how counterfactual predictions are compared: `"difference"` (default), `"ratio"`, `"lift"`, and others.
- Setting `comparison="ratio"` computes the ratio of predicted outcomes under treatment vs. control.
- Setting `comparison="lift"` computes the relative change (treatment minus control, divided by control).
- When using ratio or lift, set `hypothesis=1` to test against the null that the two predictions are identical.
- Custom comparison functions can be passed to the `comparison` argument for fully flexible analyses.
### Predictor grids {.unnumbered}
- Focal variables: for binary predictors, comparisons default to a 0-to-1 change; explicit direction can be set via `variables={"incentive": [1, 0]}`.
- For categorical predictors, `comparisons()` returns contrasts between each level and the reference level by default.
- For numeric predictors, contrasts can be specified as a fixed increment (`5`), a standard deviation (`"sd"`), specific values (`[0, 3]`), interquartile range (`"iqr"`), or full range (`"minmax"`).
- Cross-comparisons assess the joint effect of changing multiple predictors simultaneously via `cross=True`.
- Adjustment variables: by default, comparisons are computed for every row of the original dataset; alternative grids (interesting, representative, balanced) can be set via `newdata`.
### Aggregation {.unnumbered}
- `avg_comparisons()` averages unit-level comparisons across the dataset, yielding an average marginal effect or average treatment effect.
- The `by` argument computes subgroup-specific averages, e.g., `avg_comparisons(mod, variables="incentive", by="agecat")`.
- Restricting `newdata` to treated units (e.g., `dat.filter(pl.col("incentive") == 1)`) computes an average treatment effect on the treated (ATT).
- Average predictions, average counterfactual predictions, and average counterfactual comparisons are distinct quantities: the first uses observed covariates, the second replicates data under fixed focal values, and the third takes the difference of the second.
- The `variables` argument with `"sequential"` computes sequential pairwise comparisons across ordered levels of a categorical variable.
### Hypothesis tests {.unnumbered}
- The `hypothesis` argument in `avg_comparisons()` tests whether subgroup-specific average comparisons differ from each other.
- For example, `hypothesis="b0 - b2 = 0"` tests whether the average comparison in the first subgroup differs from the third.
- A large p-value means the estimated treatment effects in different subgroups are not statistically distinguishable.
- This workflow is useful for assessing treatment effect heterogeneity across subgroups.
- The syntax mirrors the hypothesis testing framework introduced in the main hypothesis chapter.
### Visualization {.unnumbered}
- `plot_comparisons()` visualizes average or conditional comparisons as point estimates with confidence intervals.
- The `by` argument creates marginal comparison plots across subgroups, e.g., `plot_comparisons(mod, variables="incentive", by="agecat")`.
- The `condition` argument creates conditional comparison plots showing how the effect of a focal variable changes with a continuous predictor.
- Multiple variables in `condition` produce faceted or color-coded plots, e.g., `condition=["distance", "agecat"]`.
- These plots help assess whether treatment effects vary systematically across predictor values.
## 7. Slopes {.unnumbered}
### Predictor grids {.unnumbered}
- The `slopes()` function computes partial derivatives (marginal effects) of the predicted outcome with respect to a focal predictor.
- In logistic regression, the slope of the probability with respect to X varies across the predictor space because of the non-linear link function.
- `slopes()` can compute slopes at specific points via `newdata=datagrid(...)`, at the mean via `newdata="mean"`, or for every observation (the default).
- When the model includes polynomials or interactions (e.g., `incentive * distance * I(distance**2)`), slopes capture the combined marginal effect.
- The result includes standard errors and confidence intervals computed via the delta method.
### Aggregation {.unnumbered}
- `avg_slopes()` averages unit-level slopes across the dataset, yielding an average marginal effect.
- The `by` argument computes subgroup-specific average slopes, e.g., `avg_slopes(mod, variables="distance", by="incentive")`.
- Average marginal effects summarize how a one-unit change in a predictor relates to the outcome on average.
- These averages are computed over the empirical distribution of covariates, which accounts for heterogeneity in the population.
- The result is a single-row (or few-row) data frame with estimate, standard error, and confidence interval.
### Hypothesis tests {.unnumbered}
- The `hypothesis` argument in `avg_slopes()` tests whether average slopes differ across subgroups.
- For example, `hypothesis="b0 - b1 = 0"` tests whether the average slope of distance differs between incentive groups.
- This tests treatment effect heterogeneity on the slope scale.
- The syntax is identical to that used in `avg_predictions()` and `avg_comparisons()`.
- A significant result suggests the relationship between the predictor and outcome differs across subgroups.
### Visualization {.unnumbered}
- `plot_predictions(mod, condition="distance")` shows the predicted outcome as a function of distance, giving context for interpreting slopes.
- `plot_slopes(mod, variables="distance", condition="distance")` shows how the marginal effect of distance varies across its own range.
- Adding a second variable to `condition` (e.g., `condition=["distance", "incentive"]`) facets or colors the slope plot by a grouping variable.
- `plot_slopes()` with the `by` argument shows average slopes for each subgroup as point estimates with confidence intervals.
- These visualizations complement numerical slope estimates by making non-linear patterns visible.
## 8. Causal inference with G-computation {.unnumbered}
### Treatment effects {.unnumbered}
- G-computation estimates causal effects by fitting an outcome model with treatment and confounders, then comparing predictions under counterfactual treatment assignments.
- The procedure: set treatment to 0 for all units, predict; set treatment to 1 for all units, predict; compare.
- `avg_predictions(mod, variables="win_big", by="win_big")` computes average counterfactual predictions under each treatment level.
- `avg_comparisons(mod, variables="win_big")` computes the average treatment effect (ATE) directly.
- Restricting `newdata` to treated or control units yields the average treatment effect on the treated (ATT) or untreated (ATU), respectively.
### Conditional average treatment effects {.unnumbered}
- `avg_comparisons(mod, variables="win_big", by="work")` estimates the conditional average treatment effect (CATE) for subgroups defined by the `work` variable.
- CATEs reveal treatment effect heterogeneity: the effect of treatment may differ across subgroups.
- The `by` argument can be set to any categorical variable in the model to explore different sources of heterogeneity.
- G-computation requires correct specification of the outcome model for valid causal inference.
- The same `avg_comparisons()` workflow used for descriptive analysis extends directly to causal inference under appropriate assumptions.
## 9. Experiments {.unnumbered}
### Covariate adjustment {.unnumbered}
- In a randomized experiment, a simple model like `outcome ~ incentive` estimates the treatment effect; `vcov="HC2"` provides heteroskedasticity-robust standard errors.
- Adding covariates (e.g., `incentive * (age + distance + hiv2004)`) can improve precision without biasing the treatment effect estimate.
- `avg_comparisons(mod, variables="incentive", vcov="HC2")` computes the average treatment effect with robust inference.
- Covariate adjustment interacts treatment with covariates, allowing the treatment effect to vary across the population.
- The `marginaleffects` workflow is the same whether or not covariates are included: fit the model, then call `avg_comparisons()`.
### Factorial experiments {.unnumbered}
- Factorial designs randomize multiple treatments simultaneously; the model includes main effects and interactions (e.g., `Y ~ Ta + Tb + Ta:Tb`).
- `plot_predictions(mod, by=["Ta", "Tb"])` visualizes predicted outcomes across all treatment combinations.
- `avg_comparisons(mod, variables="Ta", by="Tb")` estimates the effect of one treatment within levels of the other.
- `avg_comparisons(mod, variables=["Ta", "Tb"], cross=True)` estimates the joint effect of changing both treatments simultaneously.
- `hypothesis="b1 - b0 = 0"` tests whether the effect of one treatment differs across levels of the other, quantifying the interaction.
## 10. Interactions and polynomials {.unnumbered}
### Multiplicative interactions {.unnumbered}
- Categorical-by-categorical: fit `Y ~ X * M`, then use `avg_comparisons(mod, variables="X", by="M")` to estimate the effect of X within each level of M.
- Categorical-by-continuous: use `predictions()` at specific values of the continuous moderator, or `plot_predictions(mod, condition=["M", "X"])` to visualize.
- Continuous-by-continuous: use `slopes()` to compute the marginal effect of X at specific values of M, or `plot_slopes(mod, variables="X", condition="M")` to visualize.
- `hypothesis="b2 - b0 = 0"` or similar strings test whether the effect of X differs across levels or values of M.
- Multiple interactions (e.g., `X * M1 * M2`) are handled by specifying `by=["M2", "M1"]` and using composite hypothesis strings like `"(b1 - b0) - (b3 - b2) = 0"` to test higher-order interactions.
### Polynomials {.unnumbered}
- Polynomial terms (e.g., `I(X**2)`, `I(X**3)`) capture non-linear relationships; `plot_predictions(mod, condition="X", points=0.05)` overlays the fitted curve on the data.
- `slopes()` at specific values of X (e.g., `datagrid(X=[-2, 0, 2])`) reports the instantaneous marginal effect at each point.
- Polynomials can be interacted with categorical or continuous moderators (e.g., `M * (X + I(X**2) + I(X**3))`).
- `plot_predictions(mod, condition=["X", "M"], points=0.1)` visualizes how the polynomial curve differs across moderator levels.
- `slopes()` with `newdata=datagrid(M=[0,1], X=fivenum)` reports marginal effects at the five-number summary of X, separately for each moderator level.
## 13. Machine learning {.unnumbered}
### Predictions {.unnumbered}
- `fit_sklearn()` wraps a scikit-learn pipeline so it can be used with `marginaleffects` functions; it requires a function that splits data into X and y, the data, and the pipeline engine.
- The pipeline typically includes a preprocessor (e.g., `OneHotEncoder` for categorical variables) and a model (e.g., `XGBRegressor`).
- `avg_predictions(mod, by="unit_type", newdata=test)` computes average predictions by subgroup on held-out test data.
- `plot_predictions(mod, by=["bedrooms", "unit_type"])` visualizes predicted outcomes across predictor combinations.
- Counterfactual grids can be constructed with `datagrid(grid_type="counterfactual")` for machine learning models, just as for parametric models.
### Comparisons {.unnumbered}
- `avg_comparisons(mod, variables={"bedrooms": 2})` estimates the average effect of a 2-unit increase in bedrooms on predicted price.
- Cross-comparisons with `cross=True` estimate the joint effect of changing multiple predictors simultaneously.
- The `marginaleffects` workflow for machine learning models is identical to that for parametric models: fit, predict, compare, aggregate.
- This model-agnostic approach means the same interpretation tools apply whether the underlying model is a logistic regression or a gradient-boosted tree.
- Standard errors for machine learning models may require bootstrap or other resampling methods, as classical delta-method standard errors assume a parametric model.
## Roadmap {#sec-python-roadmap}
- Bayesian models.
- Mixed effects models.
- Categorical outcome models.
- Functions in the `comparison` argument of `comparisons()`.
- Rug plots in `plot_*()` functions.
- Bootstrap, simulation-based inference, and conformal prediction.
- Multiple comparison correction.
- Clustered standard errors.
- Robust standard errors and link scale predictions for logistic regression models with Statsmodels.
# Examples
## Example 1: Predictions on a counterfactual grid
```python
import polars as pl
import numpy as np
from marginaleffects import *
from plotnine import *
from statsmodels.formula.api import logit
dat = get_dataset("thornton")
dat = dat.drop_nulls(subset=["incentive"])
mod = logit("outcome ~ incentive + agecat + distance",
data=dat.to_pandas()).fit()
# Average predictions by age category
avg_predictions(mod, by="agecat")
# Average predictions on a balanced grid
avg_predictions(mod, newdata="balanced", by="agecat")
# Conditional prediction plot
plot_predictions(mod, condition=["distance", "incentive"]).show()
```
## Example 2: Average comparisons with hypothesis test
```python
from marginaleffects import *
from statsmodels.formula.api import logit
dat = get_dataset("thornton")
mod = logit("outcome ~ incentive * (agecat + distance)",
data=dat.to_pandas()).fit()
# Average treatment effect of incentive
avg_comparisons(mod, variables="incentive")
# Average treatment effect by age subgroup
avg_comparisons(mod, variables="incentive", by="agecat")
# Test whether the effect differs between age subgroups
avg_comparisons(mod, variables="incentive", by="agecat",
hypothesis="b0 - b2 = 0")
```
## Example 3: Slopes and visualization
```python
import numpy as np
from marginaleffects import *
from statsmodels.formula.api import logit
dat = get_dataset("thornton")
mod = logit(
"outcome ~ incentive * distance * I(distance**2)",
data=dat.to_pandas()).fit()
# Average marginal effect of distance
avg_slopes(mod, variables="distance")
# Average slopes by incentive subgroup
avg_slopes(mod, variables="distance", by="incentive")
# Visualize how the slope of distance varies
plot_slopes(mod, variables="distance",
condition=["distance", "incentive"]).show()
```
---
# Function Reference (R)
---
## Predictions {.unnumbered}
### Description
Outcome predicted by a fitted model on a specified scale for a given combination of values of the predictor variables, such as their observed values, their means, or factor levels (a.k.a. "reference grid").
-
predictions(): unit-level (conditional) estimates.
-
avg_predictions(): average (marginal) estimates.
The newdata argument and the datagrid() function can be used to control where statistics are evaluated in the predictor space: "at observed values", "at the mean", "at representative values", etc.
See the predictions vignette and package website for worked examples and case studies:
### Usage
predictions(
model,
newdata = NULL,
variables = NULL,
vcov = TRUE,
conf_level = 0.95,
type = NULL,
by = FALSE,
byfun = NULL,
wts = FALSE,
transform = NULL,
hypothesis = NULL,
equivalence = NULL,
df = Inf,
numderiv = "fdforward",
...
)
avg_predictions(
model,
newdata = NULL,
variables = NULL,
vcov = TRUE,
conf_level = 0.95,
type = NULL,
by = TRUE,
byfun = NULL,
wts = FALSE,
transform = NULL,
hypothesis = NULL,
equivalence = NULL,
df = Inf,
numderiv = "fdforward",
...
)
### Arguments
model |
Model object
|
newdata |
Grid of predictor values at which we evaluate predictions.
- Warning: Please avoid modifying your dataset between fitting the model and calling a
marginaleffects function. This can sometimes lead to unexpected results.
-
NULL (default): Unit-level predictions for each observed value in the dataset (empirical distribution). The dataset is retrieved using insight::get_data(), which tries to extract data from the environment. This may produce unexpected results if the original data frame has been altered since fitting the model.
- string:
- "mean": Predictions evaluated when each predictor is held at its mean or mode.
- "median": Predictions evaluated when each predictor is held at its median or mode.
- "balanced": Predictions evaluated on a balanced grid with every combination of categories and numeric variables held at their means.
- "tukey": Predictions evaluated at Tukey's 5 numbers.
- "grid": Predictions evaluated on a grid of representative numbers (Tukey's 5 numbers and unique values of categorical predictors).
-
datagrid() call to specify a custom grid of regressors. For example:
-
newdata = datagrid(cyl = c(4, 6)): cyl variable equal to 4 and 6 and other regressors fixed at their means or modes.
- See the Examples section and the
datagrid() documentation.
-
subset() call with a single argument to select a subset of the dataset used to fit the model, ex: newdata = subset(treatment == 1)
-
dplyr::filter() call with a single argument to select a subset of the dataset used to fit the model, ex: newdata = filter(treatment == 1)
|
variables |
Counterfactual variables.
- Output:
-
predictions(): The entire dataset is replicated once for each unique combination of variables, and predictions are made.
-
avg_predictions(): The entire dataset is replicated, predictions are made, and they are marginalized by variables categories.
- Warning: This can be expensive in large datasets.
- Warning: Users who need "conditional" predictions should use the
newdata argument instead of variables.
- Input:
-
NULL: computes one prediction per row of newdata
- Character vector: the dataset is replicated once of every combination of unique values of the variables identified in
variables.
- Named list: names identify the subset of variables of interest and their values. For numeric variables, the
variables argument supports functions and string shortcuts:
- A function which returns a numeric value
- Numeric vector: Contrast between the 2nd element and the 1st element of the
x vector.
- "iqr": Contrast across the interquartile range of the regressor.
- "sd": Contrast across one standard deviation around the regressor mean.
- "2sd": Contrast across two standard deviations around the regressor mean.
- "minmax": Contrast between the maximum and the minimum values of the regressor.
- "threenum": mean and 1 standard deviation on both sides
- "fivenum": Tukey's five numbers
|
vcov |
Type of uncertainty estimates to report (e.g., for robust standard errors). Acceptable values:
- FALSE: Do not compute standard errors. This can speed up computation considerably.
- TRUE: Unit-level standard errors using the default
vcov(model) variance-covariance matrix.
- String which indicates the kind of uncertainty estimates to return.
- Heteroskedasticity-consistent:
"HC", "HC0", "HC1", "HC2", "HC3", "HC4", "HC4m", "HC5". See ?sandwich::vcovHC
- Heteroskedasticity and autocorrelation consistent:
"HAC"
- Mixed-Models degrees of freedom: "satterthwaite", "kenward-roger"
- Other:
"NeweyWest", "KernHAC", "OPG". See the sandwich package documentation.
- "rsample", "boot", "fwb", and "simulation" are passed to the
method argument of the inferences() function. To customize the bootstrap or simulation process, call inferences() directly.
- One-sided formula which indicates the name of cluster variables (e.g.,
~unit_id). This formula is passed to the cluster argument of the sandwich::vcovCL function.
- Square covariance matrix
- Function which returns a covariance matrix (e.g.,
stats::vcov(model))
|
conf_level |
numeric value between 0 and 1. Confidence level to use to build a confidence interval.
|
type |
string indicates the type (scale) of the predictions used to
compute contrasts or slopes. This can differ based on the model
type, but will typically be a string such as: "response", "link", "probs",
or "zero". When an unsupported string is entered, the model-specific list of
acceptable values is returned in an error message. When type is NULL, the
first entry in the error message is used by default. See the Type section in the documentation below.
|
by |
Aggregate unit-level estimates (aka, marginalize, average over). Valid inputs:
-
FALSE: return the original unit-level estimates.
-
TRUE: aggregate estimates for each term.
- Character vector of column names in
newdata or in the data frame produced by calling the function without the by argument.
- Data frame with a
by column of group labels, and merging columns shared by newdata or the data frame produced by calling the same function without the by argument.
- See examples below.
- For more complex aggregations, you can use the
FUN argument of the hypotheses() function. See that function's documentation and the Hypothesis Test vignettes on the marginaleffects website.
|
byfun |
A function such as mean() or sum() used to aggregate
estimates within the subgroups defined by the by argument. NULL uses the
mean() function. Must accept a numeric vector and return a single numeric
value. This is sometimes used to take the sum or mean of predicted
probabilities across outcome or predictor
levels. See examples section.
|
wts |
logical, string or numeric: weights to use when computing average predictions, contrasts or slopes. These weights only affect the averaging in avg_*() or with the by argument, and not unit-level estimates. See ?weighted.mean
- string: column name of the weights variable in
newdata. When supplying a column name to wts, it is recommended to supply the original data (including the weights variable) explicitly to newdata.
- numeric: vector of length equal to the number of rows in the original data or in
newdata (if supplied).
- FALSE: Equal weights.
- TRUE: Extract weights from the fitted object with
insight::find_weights() and use them when taking weighted averages of estimates. Warning: newdata=datagrid() returns a single average weight, which is equivalent to using wts=FALSE
|
transform |
A function applied to unit-level adjusted predictions and confidence intervals just before the function returns results. For bayesian models, this function is applied to individual draws from the posterior distribution, before computing summaries.
|
hypothesis |
specify a hypothesis test or custom contrast using a number , formula, string equation, vector, matrix, or function.
- Number: The null hypothesis used in the computation of Z and p (before applying
transform).
- String: Equation to specify linear or non-linear hypothesis tests. Two-tailed tests must include an equal
= sign. One-tailed tests must start with < or >. If the terms in coef(object) uniquely identify estimates, they can be used in the formula. Otherwise, use b1, b2, etc. to identify the position of each parameter. The b* wildcard can be used to test hypotheses on all estimates. When the hypothesis string represents a two-sided equation, the estimate column holds the value of the left side minus the right side of the equation. If a named vector is used, the names are used as labels in the output. Examples:
-
hp = drat
-
hp + drat = 12
-
b1 + b2 + b3 = 0
-
b* / b1 = 1
-
<= 0
-
>= -3.5
-
b1 >= 10
- Formula:
lhs ~ rhs | group
-
lhs
-
ratio (null = 1)
-
difference (null = 0)
- Leave empty for default value
-
rhs
-
pairwise and revpairwise: pairwise differences between estimates in each row.
-
reference: differences between the estimates in each row and the estimate in the first row.
-
sequential: difference between an estimate and the estimate in the next row.
-
meandev: difference between an estimate and the mean of all estimates.
-
meanotherdev: difference between an estimate and the mean of all other estimates, excluding the current one.
-
poly: polynomial contrasts, as computed by the stats::contr.poly() function.
-
helmert: Helmert contrasts, as computed by the stats::contr.helmert() function. Contrast 2nd level to the first, 3rd to the average of the first two, and so on.
-
trt_vs_ctrl: difference between the mean of estimates (except the first) and the first estimate.
-
I(fun(x)): custom function to manipulate the vector of estimates x. The function fun() can return multiple (potentially named) estimates.
-
group (optional)
- Column name of
newdata. Conduct hypothesis tests withing subsets of the data.
- Examples:
-
~ poly
-
~ sequential | groupid
-
~ reference
-
ratio ~ pairwise
-
difference ~ pairwise | groupid
-
~ I(x - mean(x)) | groupid
-
~ I(\(x) c(a = x[1], b = mean(x[2:3]))) | groupid
- Matrix or Vector: Each column is a vector of weights. The the output is the dot product between these vectors of weights and the vector of estimates. The matrix can have column names to label the estimates.
- Function:
- Accepts an argument
x: object produced by a marginaleffects function or a data frame with column rowid and estimate
- Returns a data frame with columns
term and estimate (mandatory) and rowid (optional).
- The function can also accept optional input arguments:
newdata, by, draws.
- This function approach will not work for Bayesian models or with bootstrapping. In those cases, it is easy to use
get_draws() to extract and manipulate the draws directly.
- See the Examples section below and the vignette: https://marginaleffects.com/chapters/hypothesis.html
- Warning: When calling
predictions() with type="invlink(link)" (the default in some models), hypothesis is tested and p values are computed on the link scale.
|
equivalence |
Numeric vector of length 2: bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. For bayesian models, this report the proportion of posterior draws in the interval and the ROPE. See Details section below.
|
df |
Degrees of freedom used to compute p values and confidence intervals.
- A single numeric value between 1 and
Inf, or a numeric vector with length equal to the number of rows in the output. When df is Inf, the normal distribution is used. When df is finite, the t distribution is used.
- "residual": Calls insight::get_df to extract degrees of freedom from the model automatically.
- "satterthwaite" or "kenward-roger": Use the Satterthwaite or Kenward-Roger approximation to compute degrees of freedom in mixed effects models.
|
numderiv |
string or list of strings indicating the method to use to for the numeric differentiation used in to compute delta method standard errors.
- "fdforward": finite difference method with forward differences (default)
- "fdcenter": finite difference method with central differences
- "richardson": Richardson extrapolation method
- Extra arguments can be specified by passing a list to the
numDeriv argument, with the name of the method first and named arguments following, ex: numderiv=list("fdcenter", eps = 1e-5). When an unknown argument is used, marginaleffects prints the list of valid arguments for each method.
|
... |
Additional arguments are passed to the predict() method
supplied by the modeling package.These arguments are particularly useful
for mixed-effects or bayesian models (see the online vignettes on the
marginaleffects website). Available arguments can vary from model to
model, depending on the range of supported arguments by each modeling
package. See the "Model-Specific Arguments" section of the
?slopes documentation for a non-exhaustive list of available
arguments.
|
### Value
A data.frame with one row per estimate. This data frame is pretty-printed by default, but users can interact with it as a regular data frame, with functions like nrow(), head(), colnames(), etc. Values can be extracted using standard [,] or \$ operators, and manipulated using external packages like dplyr or data.table.
Columns may include:
-
rowid: row number of the newdata data frame
-
group: (optional) value of the grouped outcome (e.g., categorical outcome models)
-
term: the focal variable.
-
estimate: an estimate of the prediction, counterfactual comparison, or slope.
-
std.error: standard errors computed via the delta method.
-
p.value: p value associated to the estimate column. The null is determined by the hypothesis argument (0 by default).
-
s.value: Shannon information transforms of p values. See the S values vignette at https://marginaleffects.com the marginaleffects website.
-
conf.low: lower bound of the confidence (or credible) interval defined by the conf_level argument.
-
conf.high: upper bound of the confidence (or credible) interval defined by the conf_level argument.
-
predicted_lo: predicted outcome for the "low" value of the focal predictor in a counterfactual comparison.
-
predicted_hi: predicted outcome for the "high" value of the focal predictor in a counterfactual comparison.
-
p.rope.unconditional: share of posterior draws in the interval specified by the equivalence argument. This is only available for Bayesian models.
-
p.rope.conditional: share of posterior draws in the interval specified by the equivalence argument, among draws in the confidence interval. This is only available for Bayesian models.
-
rope: share of the posterior draws between conf.low and conf.high that are covered by the interval specified by the equivalence argument.
-
statistic.noninf: test statistic for non-inferiority test (when equivalence argument is used).
-
statistic.nonsup: test statistic for non-superiority test (when equivalence argument is used).
-
p.value.noninf: p-value for non-inferiority test (when equivalence argument is used).
-
p.value.nonsup: p-value for non-superiority test (when equivalence argument is used).
-
p.value.equiv: p-value for equivalence test using Two One-Sided Tests (TOST) approach (when equivalence argument is used).
See ?print.marginaleffects for printing options.
The data.frames produced by marginaleffects stores an attribute that holds many internal objects, such as the original model, data, and much other information that can be used for post-processing. This information can be inspected using the components() function.
Warning: The internal attributes retrieved by components() are not considered part of the public API of the package. Their names and contents can change without warning or notice. Users should not rely on them.
Warning: In some cases, the internal attributes used by marginaleffects() can use up a substantial amount of memory. To clear this data, use the prune() function or set options(marginaleffects_lean=TRUE).
### Functions
-
avg_predictions(): Average predictions
### Standard errors using the delta method
Standard errors for all quantities estimated by marginaleffects can be obtained via the delta method. This requires differentiating a function with respect to the coefficients in the model using a finite difference approach. In some models, the delta method standard errors can be sensitive to various aspects of the numeric differentiation strategy, including the step size. By default, the step size is set to 1e-8, or to 1e-4 times the smallest absolute model coefficient, whichever is largest.
marginaleffects can delegate numeric differentiation to the numDeriv package, which allows more flexibility. To do this, users can pass arguments to the numDeriv::jacobian function through a global option. For example:
-
options(marginaleffects_numDeriv = list(method = "simple", method.args = list(eps = 1e-6)))
-
options(marginaleffects_numDeriv = list(method = "Richardson", method.args = list(eps = 1e-5)))
-
options(marginaleffects_numDeriv = NULL)
See the "Uncertainty" chapter on the marginaleffects website for more details on the computation of standard errors, bootstrapping, and more:
https://marginaleffects.com/chapters/uncertainty.html
### Model-Specific Arguments
Some model types allow model-specific arguments to modify the nature of
marginal effects, predictions, marginal means, and contrasts. Please report
other package-specific predict() arguments on Github so we can add them to
the table below.
https://github.com/vincentarelbundock/marginaleffects/issues
|
Package | Class | Argument | Documentation |
brms | brmsfit | ndraws | brms::posterior_predict |
|
| | re_formula | brms::posterior_predict |
lme4 | merMod | re.form | lme4::predict.merMod |
|
| | allow.new.levels | lme4::predict.merMod |
glmmTMB | glmmTMB | re.form | glmmTMB::predict.glmmTMB |
|
| | allow.new.levels | glmmTMB::predict.glmmTMB |
|
| | zitype | glmmTMB::predict.glmmTMB |
mgcv | bam | exclude | mgcv::predict.bam |
|
| gam | exclude | mgcv::predict.gam |
robustlmm | rlmerMod | re.form | robustlmm::predict.rlmerMod |
|
| | allow.new.levels | robustlmm::predict.rlmerMod |
MCMCglmm | MCMCglmm | ndraws | |
sampleSelection | selection | part | sampleSelection::predict.selection |
|
|
### Bayesian posterior summaries
By default, credible intervals in bayesian models are built as equal-tailed
intervals. This can be changed to a highest density interval by setting a global
option:
options("marginaleffects_posterior_interval" = "eti")
options("marginaleffects_posterior_interval" = "hdi")
By default, the center of the posterior distribution in bayesian models is
identified by the median. Users can use a different summary function by setting a
global option:
options("marginaleffects_posterior_center" = "mean")
options("marginaleffects_posterior_center" = "median")
When estimates are averaged using the by argument, the tidy() function, or
the summary() function, the posterior distribution is marginalized twice over.
First, we take the average across units but within each iteration of the
MCMC chain, according to what the user requested in by argument or
tidy()/summary() functions. Then, we identify the center of the resulting
posterior using the function supplied to the
"marginaleffects_posterior_center" option (the median by default).
### Equivalence, Inferiority, Superiority
$\theta$ is an estimate, $\sigma_\theta$ its estimated standard error, and $[a, b]$ are the bounds of the interval supplied to the equivalence argument.
Non-inferiority:
- $H_0$: $\theta \leq a$
- $H_1$: $\theta > a$
- $t=(\theta - a)/\sigma_\theta$
- p: Upper-tail probability
Non-superiority:
- $H_0$: $\theta \geq b$
- $H_1$: $\theta < b$
- $t=(\theta - b)/\sigma_\theta$
- p: Lower-tail probability
Equivalence: Two One-Sided Tests (TOST)
- p: Maximum of the non-inferiority and non-superiority p values.
Thanks to Russell V. Lenth for the excellent emmeans package and documentation which inspired this feature.
### Order of operations
Behind the scenes, the arguments of marginaleffects functions are evaluated in this order:
-
newdata
-
variables
-
comparison and slope
-
by
-
vcov
-
hypothesis
-
transform
### Parallel computation
The slopes() and comparisons() functions can use parallelism to
speed up computation. Operations are parallelized for the computation of
standard errors, at the model coefficient level. There is always
considerable overhead when using parallel computation, mainly involved
in passing the whole dataset to the different processes. Thus, parallel
computation is most likely to be useful when the model includes many parameters
and the dataset is relatively small.
Warning: In many cases, parallel processing will not be useful at all.
To activate parallel computation, users must load the future.apply package,
call plan() function, and set a global option.
options(marginaleffects_parallel = TRUE): parallelize delta method computation of standard errors.
options(marginaleffects_parallel_inferences = TRUE): parallelize "rsample" or "fwb" bootstrap computation in inferences().
options(marginaleffects_parallel_packages = TRUE): vector of strings with the names of modeling packages used to fit the model, ex: c("survival", "splines")
For example:
library(future.apply)
plan("multisession", workers = 4)
options(marginaleffects_parallel = FALSE)
options(marginaleffects_parallel_inferences = TRUE)
options(marginaleffects_parallel_packages = c("survival", "splines"))
slopes(model)
marginaleffects altogether, you can set a global option:
options(marginaleffects_parallel = FALSE)
marginaleffects functions can be modified by setting global options.
Disable some safety checks and warnings:
-
options(marginaleffects_startup_message = FALSE)
- Disable the startup message printed on
library(marginaleffects).
-
options(marginaleffects_safe = FALSE)
- Disable safety checks and warnings.
-
options(marginaleffects_print_omit = c("p.value", "s.value"))
- Omit some columns from the printed output.
Enforce lean return objects, sans information about the original model and
data, and other ancillary attributes. Note that this will disable some
advanced post-processing features and functions like hypotheses.
options(marginaleffects_lean = TRUE)
-
marginaleffects_plot_gray: logical. If TRUE, the default color of the plot is gray. Default is FALSE.
### Types
The type argument determines the scale of the predictions used to compute quantities of interest with functions from the marginaleffects package. Admissible values for type depend on the model object. When users specify an incorrect value for type, marginaleffects will raise an informative error with a list of valid type values for the specific model object. The first entry in the list in that error message is the default type.
The invlink(link) is a special type defined by marginaleffects. It is available for some (but not all) models, and only for the predictions() function. With this link type, we first compute predictions on the link scale, then we use the inverse link function to backtransform the predictions to the response scale. This is useful for models with non-linear link functions as it can ensure that confidence intervals stay within desirable bounds, ex: 0 to 1 for a logit model. Note that an average of estimates with type="invlink(link)" will not always be equivalent to the average of estimates with type="response". This type is default when calling predictions(). It is available—but not default—when calling avg_predictions() or predictions() with the by argument.
Some of the most common type values are:
|
class | type |
|
Gam | invlink(link), response, link |
|
Gls | lp |
|
MCMCglmm | response |
|
bam | response, link |
|
bart | ev, ppd |
|
betareg | response, link, precision, quantile, variance |
|
bife | response, link |
|
bracl | probs |
|
brglmFit | response, link |
|
brmsfit | response, link, prediction, average |
|
brmultinom | probs, class |
|
clm | prob, cum.prob, linear.predictor |
|
clogit | expected, lp, risk, survival |
|
coxph | survival, expected, lp, risk |
|
coxph_weightit | survival, expected, lp, risk |
|
crch | response, location, scale, density |
|
fixest | invlink(link), response, link |
|
flexsurvreg | survival, response, mean, link, lp, linear, rmst, hazard, cumhaz |
|
gam | response, link |
|
geeglm | response, link |
|
glimML | response, link |
|
glm | invlink(link), response, link |
|
glm_weightit | invlink(link), probs, response, lp, link |
|
glmerMod | response, link |
|
glmgee | response |
|
glmmPQL | response, link |
|
glmmTMB | response, link, conditional, zprob, zlink, disp |
|
glmrob | response, link |
|
glmx | response |
|
hetprob | pr, xb |
|
hurdle | response, prob, count, zero |
|
hxlr | location, cumprob, scale, density |
|
iv_robust | response |
|
ivpml | pr, xb |
|
ivreg | response |
|
lda | class, posterior |
|
lm | response |
|
lm_robust | response |
|
lmerMod | response |
|
lmerModLmerTest | response |
|
lmrob | response |
|
lrm | fitted, lp, mean |
|
mblogit | response, latent, link |
|
mclogit | response, latent, link |
|
mhurdle | E, Ep, p |
|
model_fit | numeric, prob, class |
|
multinom | probs, latent |
|
multinom_weightit | probs, response, mean |
|
mvgam | response, link, expected, detection, latent_N |
|
negbin | invlink(link), response, link |
|
ols | lp |
|
oohbchoice | probability, utility |
|
ordinal_weightit | probs, response, link, lp, mean |
|
orm | fitted, mean, lp |
|
polr | probs |
|
rendo.base | response, link |
|
rlm | response |
|
selection | response, link, unconditional, conditional |
|
speedglm | response, link |
|
speedlm | response |
|
stanreg | response, link |
|
survreg | response, link, quantile |
|
svyglm | response, link |
|
svyolr | probs |
|
tobit | response, link |
|
tobit1 | expvalue, linpred, prob |
|
workflow | numeric, prob, class |
|
zeroinfl | response, prob, count, zero |
|
|
### References
- Arel-Bundock V, Greifer N, Heiss A (2024). “How to Interpret Statistical Models Using marginaleffects for R and Python.” Journal of Statistical Software, 111(9), 1-32. doi:10.18637/jss.v111.i09 [doi:10.18637/jss.v111.i09](https://doi.org/10.18637/jss.v111.i09)
- Arel-Bundock (2026). "Model to Meaning: How to interpret statistical models in R and Python." CRC Press. https://routledge.com/9781032908724
- Greenland S. 2019. "Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values." The American Statistician. 73(S1): 106–114.
- Cole, Stephen R, Jessie K Edwards, and Sander Greenland. 2020. "Surprise!" American Journal of Epidemiology 190 (2): 191–93. [doi:10.1093/aje/kwaa136](https://doi.org/10.1093/aje/kwaa136)
### Examples
```{r, warning=FALSE, message=FALSE, eval=FALSE}
library("marginaleffects")
library("marginaleffects")
# Adjusted Prediction for every row of the original dataset
mod <- lm(mpg ~ hp + factor(cyl), data = mtcars)
pred <- predictions(mod)
head(pred)
# Adjusted Predictions at User-Specified Values of the Regressors
predictions(mod, newdata = datagrid(hp = c(100, 120), cyl = 4))
m <- lm(mpg ~ hp + drat + factor(cyl) + factor(am), data = mtcars)
predictions(m, newdata = datagrid(FUN_factor = unique, FUN_numeric = median))
# Average Adjusted Predictions (AAP)
library(dplyr)
mod <- lm(mpg ~ hp * am * vs, mtcars)
avg_predictions(mod)
predictions(mod, by = "am")
# Conditional Adjusted Predictions
plot_predictions(mod, condition = "hp")
# Counterfactual predictions with the `variables` argument
# the `mtcars` dataset has 32 rows
mod <- lm(mpg ~ hp + am, data = mtcars)
p <- predictions(mod)
head(p)
nrow(p)
# average counterfactual predictions
avg_predictions(mod, variables = "am")
# counterfactual predictions obtained by replicating the entire for different
# values of the predictors
p <- predictions(mod, variables = list(hp = c(90, 110)))
nrow(p)
# hypothesis test: is the prediction in the 1st row equal to the prediction in the 2nd row
mod <- lm(mpg ~ wt + drat, data = mtcars)
predictions(
mod,
newdata = datagrid(wt = 2:3),
hypothesis = "b1 = b2")
# same hypothesis test using row indices
predictions(
mod,
newdata = datagrid(wt = 2:3),
hypothesis = "b1 - b2 = 0")
# same hypothesis test using numeric vector of weights
predictions(
mod,
newdata = datagrid(wt = 2:3),
hypothesis = c(1, -1))
# two custom contrasts using a matrix of weights
lc <- matrix(
c(
1, -1,
2, 3),
ncol = 2)
predictions(
mod,
newdata = datagrid(wt = 2:3),
hypothesis = lc)
# `by` argument
mod <- lm(mpg ~ hp * am * vs, data = mtcars)
predictions(mod, by = c("am", "vs"))
library(nnet)
nom <- multinom(factor(gear) ~ mpg + am * vs, data = mtcars, trace = FALSE)
# first 5 raw predictions
p <- predictions(nom, type = "probs")
head(p)
# average predictions
avg_predictions(nom, type = "probs", by = "group")
by <- data.frame(
group = c("3", "4", "5"),
by = c("3,4", "3,4", "5"))
predictions(nom, type = "probs", by = by)
# sum of predicted probabilities for combined response levels
mod <- multinom(factor(cyl) ~ mpg + am, data = mtcars, trace = FALSE)
by <- data.frame(
by = c("4,6", "4,6", "8"),
group = as.character(c(4, 6, 8)))
predictions(mod, newdata = "mean", byfun = sum, by = by)
```
---
## Comparisons Between Predictions Made With Different Regressor Values {.unnumbered}
### Description
Predict the outcome variable at different regressor values (e.g., college
graduates vs. others), and compare those predictions by computing a difference,
ratio, or some other function. comparisons() can return many quantities of
interest, such as contrasts, differences, risk ratios, changes in log odds, lift,
slopes, elasticities, etc.
-
comparisons(): unit-level (conditional) estimates.
-
avg_comparisons(): average (marginal) estimates.
variables identifies the focal regressors whose "effect" we are interested in. comparison determines how predictions with different regressor values are compared (difference, ratio, odds, etc.). The newdata argument and the datagrid() function control where statistics are evaluated in the predictor space: "at observed values", "at the mean", "at representative values", etc.
See the comparisons chapter on the package website for worked examples and case studies:
### Usage
comparisons(
model,
newdata = NULL,
variables = NULL,
comparison = "difference",
type = NULL,
vcov = TRUE,
by = FALSE,
conf_level = 0.95,
transform = NULL,
cross = FALSE,
wts = FALSE,
hypothesis = NULL,
equivalence = NULL,
df = Inf,
eps = NULL,
numderiv = "fdforward",
...
)
avg_comparisons(
model,
newdata = NULL,
variables = NULL,
type = NULL,
vcov = TRUE,
by = TRUE,
conf_level = 0.95,
comparison = "difference",
transform = NULL,
cross = FALSE,
wts = FALSE,
hypothesis = NULL,
equivalence = NULL,
df = Inf,
eps = NULL,
numderiv = "fdforward",
...
)
### Arguments
model |
Model object
|
newdata |
Grid of predictor values at which we evaluate the comparisons.
- Warning: Avoid modifying your dataset between fitting the model and calling a
marginaleffects function. This can sometimes lead to unexpected results.
-
NULL (default): Unit-level contrasts for each observed value in the dataset (empirical distribution). The dataset is retrieved using insight::get_data(), which tries to extract data from the environment. This may produce unexpected results if the original data frame has been altered since fitting the model.
- data frame: Unit-level contrasts for each row of the
newdata data frame.
- string:
- "mean": Contrasts at the Mean. Contrasts when each predictor is held at its mean or mode.
- "median": Contrasts at the Median. Contrasts when each predictor is held at its median or mode.
- "balanced": Contrasts evaluated on a balanced grid with every combination of categories and numeric variables held at their means.
- "tukey": Contrasts at Tukey's 5 numbers.
- "grid": Contrasts on a grid of representative numbers (Tukey's 5 numbers and unique values of categorical predictors).
-
datagrid() call to specify a custom grid of regressors. For example:
-
newdata = datagrid(cyl = c(4, 6)): cyl variable equal to 4 and 6 and other regressors fixed at their means or modes.
-
newdata = datagrid(mpg = fivenum): mpg variable held at Tukey's five numbers (using the fivenum function), and other regressors fixed at their means or modes.
- See the Examples section and the datagrid documentation.
-
subset() call with a single argument to select a subset of the dataset used to fit the model, ex: newdata = subset(treatment == 1)
-
dplyr::filter() call with a single argument to select a subset of the dataset used to fit the model, ex: newdata = filter(treatment == 1)
|
variables |
Focal variables
-
NULL: compute comparisons for all the variables in the model object (can be slow).
- Character vector: subset of variables (usually faster).
- Named list: names identify the subset of variables of interest, and values define the type of contrast to compute. Acceptable values depend on the variable type:
- Factor or character variables:
- "reference": Each factor level is compared to the factor reference (base) level
- "all": All combinations of observed levels
- "sequential": Each factor level is compared to the previous factor level
- "pairwise": Each factor level is compared to all other levels
- "minmax": The highest and lowest levels of a factor.
- "revpairwise", "revreference", "revsequential": inverse of the corresponding hypotheses.
- Vector of length 2 with the two values to compare.
- Data frame with the same number of rows as
newdata, with two columns of "lo" and "hi" values to compare.
- Function that accepts a vector and returns a data frame with two columns of "lo" and "hi" values to compare. See examples below.
- Logical variables:
- NULL: contrast between TRUE and FALSE
- Data frame with the same number of rows as
newdata, with two columns of "lo" and "hi" values to compare.
- Function that accepts a vector and returns a data frame with two columns of "lo" and "hi" values to compare. See examples below.
- Numeric variables:
- Numeric of length 1: Forward contrast for a gap of
x, computed between the observed value and the observed value plus x. Users can set a global option to get a "center" or "backward" contrast instead: options(marginaleffects_contrast_direction="center")
- Numeric vector of length 2: Contrast between the largest and the smallest elements of the
x vector.
- Data frame with the same number of rows as
newdata, with two columns of "lo" and "hi" values to compare.
- Function that accepts a vector and returns a data frame with two columns of "lo" and "hi" values to compare. See examples below.
- "iqr": Contrast across the interquartile range of the regressor.
- "sd": Contrast across one standard deviation around the regressor mean.
- "2sd": Contrast across two standard deviations around the regressor mean.
- "minmax": Contrast between the maximum and the minimum values of the regressor.
- Examples:
-
variables = list(gear = "pairwise", hp = 10)
-
variables = list(gear = "sequential", hp = c(100, 120))
-
variables = list(hp = function(x) data.frame(low = x - 5, high = x + 10))
- See the Examples section below for more.
|
comparison |
How should pairs of predictions be compared? Difference, ratio, odds ratio, or user-defined functions.
- string: shortcuts to common contrast functions. Supported shortcuts strings: difference, differenceavg, differenceavgwts, dydx, eyex, eydx, dyex, dydxavg, eyexavg, eydxavg, dyexavg, dydxavgwts, eyexavgwts, eydxavgwts, dyexavgwts, ratio, ratioavg, ratioavgwts, lnratio, lnratioavg, lnratioavgwts, lnor, lnoravg, lnoravgwts, lift, liftavg, liftavgwts, expdydx, expdydxavg, expdydxavgwts
- function: accept two equal-length numeric vectors of adjusted predictions (
hi and lo) and returns a vector of contrasts of the same length, or a unique numeric value.
- See the "Comparison functions" section below for a list of common transformations and the definitions of available shortcuts.
|
type |
string indicates the type (scale) of the predictions used to
compute contrasts or slopes. This can differ based on the model
type, but will typically be a string such as: "response", "link", "probs",
or "zero". When an unsupported string is entered, the model-specific list of
acceptable values is returned in an error message. When type is NULL, the
first entry in the error message is used by default. See the Type section in the documentation below.
|
vcov |
Type of uncertainty estimates to report (e.g., for robust standard errors). Acceptable values:
- FALSE: Do not compute standard errors. This can speed up computation considerably.
- TRUE: Unit-level standard errors using the default
vcov(model) variance-covariance matrix.
- String which indicates the kind of uncertainty estimates to return.
- Heteroskedasticity-consistent:
"HC", "HC0", "HC1", "HC2", "HC3", "HC4", "HC4m", "HC5". See ?sandwich::vcovHC
- Heteroskedasticity and autocorrelation consistent:
"HAC"
- Mixed-Models degrees of freedom: "satterthwaite", "kenward-roger"
- Other:
"NeweyWest", "KernHAC", "OPG". See the sandwich package documentation.
- "rsample", "boot", "fwb", and "simulation" are passed to the
method argument of the inferences() function. To customize the bootstrap or simulation process, call inferences() directly.
- One-sided formula which indicates the name of cluster variables (e.g.,
~unit_id). This formula is passed to the cluster argument of the sandwich::vcovCL function.
- Square covariance matrix
- Function which returns a covariance matrix (e.g.,
stats::vcov(model))
|
by |
Aggregate unit-level estimates (aka, marginalize, average over). Valid inputs:
-
FALSE: return the original unit-level estimates.
-
TRUE: aggregate estimates for each term.
- Character vector of column names in
newdata or in the data frame produced by calling the function without the by argument.
- Data frame with a
by column of group labels, and merging columns shared by newdata or the data frame produced by calling the same function without the by argument.
- See examples below.
- For more complex aggregations, you can use the
FUN argument of the hypotheses() function. See that function's documentation and the Hypothesis Test vignettes on the marginaleffects website.
|
conf_level |
numeric value between 0 and 1. Confidence level to use to build a confidence interval.
|
transform |
string or function. Transformation applied to unit-level estimates and confidence intervals just before the function returns results. Functions must accept a vector and return a vector of the same length. Support string shortcuts: "exp", "ln"
|
cross |
-
FALSE: Contrasts represent the change in adjusted predictions when one predictor changes and all other variables are held constant.
-
TRUE: Contrasts represent the changes in adjusted predictions when all the predictors specified in the variables argument are manipulated simultaneously (a "cross-contrast").
|
wts |
logical, string or numeric: weights to use when computing average predictions, contrasts or slopes. These weights only affect the averaging in avg_*() or with the by argument, and not unit-level estimates. See ?weighted.mean
- string: column name of the weights variable in
newdata. When supplying a column name to wts, it is recommended to supply the original data (including the weights variable) explicitly to newdata.
- numeric: vector of length equal to the number of rows in the original data or in
newdata (if supplied).
- FALSE: Equal weights.
- TRUE: Extract weights from the fitted object with
insight::find_weights() and use them when taking weighted averages of estimates. Warning: newdata=datagrid() returns a single average weight, which is equivalent to using wts=FALSE
|
hypothesis |
specify a hypothesis test or custom contrast using a number , formula, string equation, vector, matrix, or function.
- Number: The null hypothesis used in the computation of Z and p (before applying
transform).
- String: Equation to specify linear or non-linear hypothesis tests. Two-tailed tests must include an equal
= sign. One-tailed tests must start with < or >. If the terms in coef(object) uniquely identify estimates, they can be used in the formula. Otherwise, use b1, b2, etc. to identify the position of each parameter. The b* wildcard can be used to test hypotheses on all estimates. When the hypothesis string represents a two-sided equation, the estimate column holds the value of the left side minus the right side of the equation. If a named vector is used, the names are used as labels in the output. Examples:
-
hp = drat
-
hp + drat = 12
-
b1 + b2 + b3 = 0
-
b* / b1 = 1
-
<= 0
-
>= -3.5
-
b1 >= 10
- Formula:
lhs ~ rhs | group
-
lhs
-
ratio (null = 1)
-
difference (null = 0)
- Leave empty for default value
-
rhs
-
pairwise and revpairwise: pairwise differences between estimates in each row.
-
reference: differences between the estimates in each row and the estimate in the first row.
-
sequential: difference between an estimate and the estimate in the next row.
-
meandev: difference between an estimate and the mean of all estimates.
-
meanotherdev: difference between an estimate and the mean of all other estimates, excluding the current one.
-
poly: polynomial contrasts, as computed by the stats::contr.poly() function.
-
helmert: Helmert contrasts, as computed by the stats::contr.helmert() function. Contrast 2nd level to the first, 3rd to the average of the first two, and so on.
-
trt_vs_ctrl: difference between the mean of estimates (except the first) and the first estimate.
-
I(fun(x)): custom function to manipulate the vector of estimates x. The function fun() can return multiple (potentially named) estimates.
-
group (optional)
- Column name of
newdata. Conduct hypothesis tests withing subsets of the data.
- Examples:
-
~ poly
-
~ sequential | groupid
-
~ reference
-
ratio ~ pairwise
-
difference ~ pairwise | groupid
-
~ I(x - mean(x)) | groupid
-
~ I(\(x) c(a = x[1], b = mean(x[2:3]))) | groupid
- Matrix or Vector: Each column is a vector of weights. The the output is the dot product between these vectors of weights and the vector of estimates. The matrix can have column names to label the estimates.
- Function:
- Accepts an argument
x: object produced by a marginaleffects function or a data frame with column rowid and estimate
- Returns a data frame with columns
term and estimate (mandatory) and rowid (optional).
- The function can also accept optional input arguments:
newdata, by, draws.
- This function approach will not work for Bayesian models or with bootstrapping. In those cases, it is easy to use
get_draws() to extract and manipulate the draws directly.
- See the Examples section below and the vignette: https://marginaleffects.com/chapters/hypothesis.html
- Warning: When calling
predictions() with type="invlink(link)" (the default in some models), hypothesis is tested and p values are computed on the link scale.
|
equivalence |
Numeric vector of length 2: bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. For bayesian models, this report the proportion of posterior draws in the interval and the ROPE. See Details section below.
|
df |
Degrees of freedom used to compute p values and confidence intervals.
- A single numeric value between 1 and
Inf, or a numeric vector with length equal to the number of rows in the output. When df is Inf, the normal distribution is used. When df is finite, the t distribution is used.
- "residual": Calls insight::get_df to extract degrees of freedom from the model automatically.
- "satterthwaite" or "kenward-roger": Use the Satterthwaite or Kenward-Roger approximation to compute degrees of freedom in mixed effects models.
|
eps |
NULL or numeric value which determines the step size to use when
calculating numerical derivatives: (f(x+eps)-f(x))/eps. When eps is
NULL, the step size is 0.0001 multiplied by the difference between
the maximum and minimum values of the variable with respect to which we
are taking the derivative. Changing eps may be necessary to avoid
numerical problems in certain models.
|
numderiv |
string or list of strings indicating the method to use to for the numeric differentiation used in to compute delta method standard errors.
- "fdforward": finite difference method with forward differences (default)
- "fdcenter": finite difference method with central differences
- "richardson": Richardson extrapolation method
- Extra arguments can be specified by passing a list to the
numDeriv argument, with the name of the method first and named arguments following, ex: numderiv=list("fdcenter", eps = 1e-5). When an unknown argument is used, marginaleffects prints the list of valid arguments for each method.
|
... |
Additional arguments are passed to the predict() method
supplied by the modeling package.These arguments are particularly useful
for mixed-effects or bayesian models (see the online vignettes on the
marginaleffects website). Available arguments can vary from model to
model, depending on the range of supported arguments by each modeling
package. See the "Model-Specific Arguments" section of the
?slopes documentation for a non-exhaustive list of available
arguments.
|
### Value
A data.frame with one row per estimate. This data frame is pretty-printed by default, but users can interact with it as a regular data frame, with functions like nrow(), head(), colnames(), etc. Values can be extracted using standard [,] or \$ operators, and manipulated using external packages like dplyr or data.table.
Columns may include:
-
rowid: row number of the newdata data frame
-
group: (optional) value of the grouped outcome (e.g., categorical outcome models)
-
term: the focal variable.
-
estimate: an estimate of the prediction, counterfactual comparison, or slope.
-
std.error: standard errors computed via the delta method.
-
p.value: p value associated to the estimate column. The null is determined by the hypothesis argument (0 by default).
-
s.value: Shannon information transforms of p values. See the S values vignette at https://marginaleffects.com the marginaleffects website.
-
conf.low: lower bound of the confidence (or credible) interval defined by the conf_level argument.
-
conf.high: upper bound of the confidence (or credible) interval defined by the conf_level argument.
-
predicted_lo: predicted outcome for the "low" value of the focal predictor in a counterfactual comparison.
-
predicted_hi: predicted outcome for the "high" value of the focal predictor in a counterfactual comparison.
-
p.rope.unconditional: share of posterior draws in the interval specified by the equivalence argument. This is only available for Bayesian models.
-
p.rope.conditional: share of posterior draws in the interval specified by the equivalence argument, among draws in the confidence interval. This is only available for Bayesian models.
-
rope: share of the posterior draws between conf.low and conf.high that are covered by the interval specified by the equivalence argument.
-
statistic.noninf: test statistic for non-inferiority test (when equivalence argument is used).
-
statistic.nonsup: test statistic for non-superiority test (when equivalence argument is used).
-
p.value.noninf: p-value for non-inferiority test (when equivalence argument is used).
-
p.value.nonsup: p-value for non-superiority test (when equivalence argument is used).
-
p.value.equiv: p-value for equivalence test using Two One-Sided Tests (TOST) approach (when equivalence argument is used).
See ?print.marginaleffects for printing options.
The data.frames produced by marginaleffects stores an attribute that holds many internal objects, such as the original model, data, and much other information that can be used for post-processing. This information can be inspected using the components() function.
Warning: The internal attributes retrieved by components() are not considered part of the public API of the package. Their names and contents can change without warning or notice. Users should not rely on them.
Warning: In some cases, the internal attributes used by marginaleffects() can use up a substantial amount of memory. To clear this data, use the prune() function or set options(marginaleffects_lean=TRUE).
### Functions
-
avg_comparisons(): Average comparisons
### Standard errors using the delta method
Standard errors for all quantities estimated by marginaleffects can be obtained via the delta method. This requires differentiating a function with respect to the coefficients in the model using a finite difference approach. In some models, the delta method standard errors can be sensitive to various aspects of the numeric differentiation strategy, including the step size. By default, the step size is set to 1e-8, or to 1e-4 times the smallest absolute model coefficient, whichever is largest.
marginaleffects can delegate numeric differentiation to the numDeriv package, which allows more flexibility. To do this, users can pass arguments to the numDeriv::jacobian function through a global option. For example:
-
options(marginaleffects_numDeriv = list(method = "simple", method.args = list(eps = 1e-6)))
-
options(marginaleffects_numDeriv = list(method = "Richardson", method.args = list(eps = 1e-5)))
-
options(marginaleffects_numDeriv = NULL)
See the "Uncertainty" chapter on the marginaleffects website for more details on the computation of standard errors, bootstrapping, and more:
https://marginaleffects.com/chapters/uncertainty.html
### Model-Specific Arguments
Some model types allow model-specific arguments to modify the nature of
marginal effects, predictions, marginal means, and contrasts. Please report
other package-specific predict() arguments on Github so we can add them to
the table below.
https://github.com/vincentarelbundock/marginaleffects/issues
|
Package | Class | Argument | Documentation |
brms | brmsfit | ndraws | brms::posterior_predict |
|
| | re_formula | brms::posterior_predict |
lme4 | merMod | re.form | lme4::predict.merMod |
|
| | allow.new.levels | lme4::predict.merMod |
glmmTMB | glmmTMB | re.form | glmmTMB::predict.glmmTMB |
|
| | allow.new.levels | glmmTMB::predict.glmmTMB |
|
| | zitype | glmmTMB::predict.glmmTMB |
mgcv | bam | exclude | mgcv::predict.bam |
|
| gam | exclude | mgcv::predict.gam |
robustlmm | rlmerMod | re.form | robustlmm::predict.rlmerMod |
|
| | allow.new.levels | robustlmm::predict.rlmerMod |
MCMCglmm | MCMCglmm | ndraws | |
sampleSelection | selection | part | sampleSelection::predict.selection |
|
|
### Comparison functions
Each of the quantities computed by comparisons() can be defined as a function of these quantities:
-
hi: vector of predictions for the "high" side of the contrast.
-
lo: vector of predictions for the "low" side of the contrast.
-
y: predictions for the original data.
-
x: focal predictor in the original data.
-
w: weights
For example, the "lift" of a binary predictor is a popular quantity of
interest, defined as the difference between predictions when the focal
predictor $X = 1$, and predictions when the focal predictor is
$X = 0$, normalized by the starting point. Or:
$\frac{\hat{Y}_{X=1} - \hat{Y}_{X=0}}{\hat{Y}_{X=0}}$
When, the argument is set to comparison="lift", marginaleffects will compute the quantity using this function:
function(hi, lo) { (hi - lo) / lo }
Users can supply custom functions to the comparison argument, or use one of the many shortcuts available for common quantities of interest:
|
Shortcut | Function |
|
difference | function (hi, lo) hi - lo |
|
differenceavg | function (hi, lo) mean(hi - lo) |
|
dydx | function (hi, lo, eps) (hi - lo)/eps |
|
eyex | function (hi, lo, eps, y, x) (hi - lo)/eps * (x/y) |
|
eydx | function (hi, lo, eps, y, x) ((hi - lo)/eps)/y |
|
dyex | function (hi, lo, eps, x) ((hi - lo)/eps) * x |
|
dydxavg | function (hi, lo, eps) mean((hi - lo)/eps) |
|
eyexavg | function (hi, lo, eps, y, x) mean((hi - lo)/eps * (x/y)) |
|
eydxavg | function (hi, lo, eps, y, x) mean(((hi - lo)/eps)/y) |
|
dyexavg | function (hi, lo, eps, x) mean(((hi - lo)/eps) * x) |
|
ratio | function (hi, lo) hi/lo |
|
ratioavg | function (hi, lo) mean(hi)/mean(lo) |
|
lnratio | function (hi, lo) log(hi/lo) |
|
lnratioavg | function (hi, lo) log(mean(hi)/mean(lo)) |
|
lnor | function (hi, lo) log((hi/(1 - hi))/(lo/(1 - lo))) |
|
lnoravg | function (hi, lo) log((mean(hi)/(1 - mean(hi)))/(mean(lo)/(1 - mean(lo)))) |
|
lift | function (hi, lo) (hi - lo)/lo |
|
liftavg | function (hi, lo) (mean(hi - lo))/mean(lo) |
|
expdydx | function (hi, lo, eps) ((exp(hi) - exp(lo))/exp(eps))/eps |
|
expdydxavg | function (hi, lo, eps) mean(((exp(hi) - exp(lo))/exp(eps))/eps) |
|
|
### Bayesian posterior summaries
By default, credible intervals in bayesian models are built as equal-tailed
intervals. This can be changed to a highest density interval by setting a global
option:
options("marginaleffects_posterior_interval" = "eti")
options("marginaleffects_posterior_interval" = "hdi")
By default, the center of the posterior distribution in bayesian models is
identified by the median. Users can use a different summary function by setting a
global option:
options("marginaleffects_posterior_center" = "mean")
options("marginaleffects_posterior_center" = "median")
When estimates are averaged using the by argument, the tidy() function, or
the summary() function, the posterior distribution is marginalized twice over.
First, we take the average across units but within each iteration of the
MCMC chain, according to what the user requested in by argument or
tidy()/summary() functions. Then, we identify the center of the resulting
posterior using the function supplied to the
"marginaleffects_posterior_center" option (the median by default).
### Equivalence, Inferiority, Superiority
$\theta$ is an estimate, $\sigma_\theta$ its estimated standard error, and $[a, b]$ are the bounds of the interval supplied to the equivalence argument.
Non-inferiority:
- $H_0$: $\theta \leq a$
- $H_1$: $\theta > a$
- $t=(\theta - a)/\sigma_\theta$
- p: Upper-tail probability
Non-superiority:
- $H_0$: $\theta \geq b$
- $H_1$: $\theta < b$
- $t=(\theta - b)/\sigma_\theta$
- p: Lower-tail probability
Equivalence: Two One-Sided Tests (TOST)
- p: Maximum of the non-inferiority and non-superiority p values.
Thanks to Russell V. Lenth for the excellent emmeans package and documentation which inspired this feature.
### Order of operations
Behind the scenes, the arguments of marginaleffects functions are evaluated in this order:
-
newdata
-
variables
-
comparison and slope
-
by
-
vcov
-
hypothesis
-
transform
### Parallel computation
The slopes() and comparisons() functions can use parallelism to
speed up computation. Operations are parallelized for the computation of
standard errors, at the model coefficient level. There is always
considerable overhead when using parallel computation, mainly involved
in passing the whole dataset to the different processes. Thus, parallel
computation is most likely to be useful when the model includes many parameters
and the dataset is relatively small.
Warning: In many cases, parallel processing will not be useful at all.
To activate parallel computation, users must load the future.apply package,
call plan() function, and set a global option.
options(marginaleffects_parallel = TRUE): parallelize delta method computation of standard errors.
options(marginaleffects_parallel_inferences = TRUE): parallelize "rsample" or "fwb" bootstrap computation in inferences().
options(marginaleffects_parallel_packages = TRUE): vector of strings with the names of modeling packages used to fit the model, ex: c("survival", "splines")
For example:
library(future.apply)
plan("multisession", workers = 4)
options(marginaleffects_parallel = FALSE)
options(marginaleffects_parallel_inferences = TRUE)
options(marginaleffects_parallel_packages = c("survival", "splines"))
slopes(model)
marginaleffects altogether, you can set a global option:
options(marginaleffects_parallel = FALSE)
marginaleffects functions can be modified by setting global options.
Disable some safety checks and warnings:
-
options(marginaleffects_startup_message = FALSE)
- Disable the startup message printed on
library(marginaleffects).
-
options(marginaleffects_safe = FALSE)
- Disable safety checks and warnings.
-
options(marginaleffects_print_omit = c("p.value", "s.value"))
- Omit some columns from the printed output.
Enforce lean return objects, sans information about the original model and
data, and other ancillary attributes. Note that this will disable some
advanced post-processing features and functions like hypotheses.
options(marginaleffects_lean = TRUE)
-
marginaleffects_plot_gray: logical. If TRUE, the default color of the plot is gray. Default is FALSE.
### Types
The type argument determines the scale of the predictions used to compute quantities of interest with functions from the marginaleffects package. Admissible values for type depend on the model object. When users specify an incorrect value for type, marginaleffects will raise an informative error with a list of valid type values for the specific model object. The first entry in the list in that error message is the default type.
The invlink(link) is a special type defined by marginaleffects. It is available for some (but not all) models, and only for the predictions() function. With this link type, we first compute predictions on the link scale, then we use the inverse link function to backtransform the predictions to the response scale. This is useful for models with non-linear link functions as it can ensure that confidence intervals stay within desirable bounds, ex: 0 to 1 for a logit model. Note that an average of estimates with type="invlink(link)" will not always be equivalent to the average of estimates with type="response". This type is default when calling predictions(). It is available—but not default—when calling avg_predictions() or predictions() with the by argument.
Some of the most common type values are:
|
class | type |
|
Gam | invlink(link), response, link |
|
Gls | lp |
|
MCMCglmm | response |
|
bam | response, link |
|
bart | ev, ppd |
|
betareg | response, link, precision, quantile, variance |
|
bife | response, link |
|
bracl | probs |
|
brglmFit | response, link |
|
brmsfit | response, link, prediction, average |
|
brmultinom | probs, class |
|
clm | prob, cum.prob, linear.predictor |
|
clogit | expected, lp, risk, survival |
|
coxph | survival, expected, lp, risk |
|
coxph_weightit | survival, expected, lp, risk |
|
crch | response, location, scale, density |
|
fixest | invlink(link), response, link |
|
flexsurvreg | survival, response, mean, link, lp, linear, rmst, hazard, cumhaz |
|
gam | response, link |
|
geeglm | response, link |
|
glimML | response, link |
|
glm | invlink(link), response, link |
|
glm_weightit | invlink(link), probs, response, lp, link |
|
glmerMod | response, link |
|
glmgee | response |
|
glmmPQL | response, link |
|
glmmTMB | response, link, conditional, zprob, zlink, disp |
|
glmrob | response, link |
|
glmx | response |
|
hetprob | pr, xb |
|
hurdle | response, prob, count, zero |
|
hxlr | location, cumprob, scale, density |
|
iv_robust | response |
|
ivpml | pr, xb |
|
ivreg | response |
|
lda | class, posterior |
|
lm | response |
|
lm_robust | response |
|
lmerMod | response |
|
lmerModLmerTest | response |
|
lmrob | response |
|
lrm | fitted, lp, mean |
|
mblogit | response, latent, link |
|
mclogit | response, latent, link |
|
mhurdle | E, Ep, p |
|
model_fit | numeric, prob, class |
|
multinom | probs, latent |
|
multinom_weightit | probs, response, mean |
|
mvgam | response, link, expected, detection, latent_N |
|
negbin | invlink(link), response, link |
|
ols | lp |
|
oohbchoice | probability, utility |
|
ordinal_weightit | probs, response, link, lp, mean |
|
orm | fitted, mean, lp |
|
polr | probs |
|
rendo.base | response, link |
|
rlm | response |
|
selection | response, link, unconditional, conditional |
|
speedglm | response, link |
|
speedlm | response |
|
stanreg | response, link |
|
survreg | response, link, quantile |
|
svyglm | response, link |
|
svyolr | probs |
|
tobit | response, link |
|
tobit1 | expvalue, linpred, prob |
|
workflow | numeric, prob, class |
|
zeroinfl | response, prob, count, zero |
|
|
### References
- Arel-Bundock V, Greifer N, Heiss A (2024). “How to Interpret Statistical Models Using marginaleffects for R and Python.” Journal of Statistical Software, 111(9), 1-32. doi:10.18637/jss.v111.i09 [doi:10.18637/jss.v111.i09](https://doi.org/10.18637/jss.v111.i09)
- Arel-Bundock (2026). "Model to Meaning: How to interpret statistical models in R and Python." CRC Press. https://routledge.com/9781032908724
- Greenland S. 2019. "Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values." The American Statistician. 73(S1): 106–114.
- Cole, Stephen R, Jessie K Edwards, and Sander Greenland. 2020. "Surprise!" American Journal of Epidemiology 190 (2): 191–93. [doi:10.1093/aje/kwaa136](https://doi.org/10.1093/aje/kwaa136)
### Examples
```{r, warning=FALSE, message=FALSE, eval=FALSE}
library("marginaleffects")
# Linear model
tmp <- mtcars
tmp$am <- as.logical(tmp$am)
mod <- lm(mpg ~ am + factor(cyl), tmp)
avg_comparisons(mod, variables = list(cyl = "reference"))
avg_comparisons(mod, variables = list(cyl = "sequential"))
avg_comparisons(mod, variables = list(cyl = "pairwise"))
# GLM with different scale types
mod <- glm(am ~ factor(gear), data = mtcars)
avg_comparisons(mod, type = "response")
avg_comparisons(mod, type = "link")
# Contrasts at the mean
comparisons(mod, newdata = "mean")
# Contrasts between marginal means
comparisons(mod, newdata = "balanced")
# Contrasts at user-specified values
comparisons(mod, newdata = datagrid(am = 0, gear = tmp$gear))
comparisons(mod, newdata = datagrid(am = unique, gear = max))
m <- lm(mpg ~ hp + drat + factor(cyl) + factor(am), data = mtcars)
comparisons(m, variables = "hp", newdata = datagrid(FUN_factor = unique, FUN_numeric = median))
# Numeric contrasts
mod <- lm(mpg ~ hp, data = mtcars)
avg_comparisons(mod, variables = list(hp = 1))
avg_comparisons(mod, variables = list(hp = 5))
avg_comparisons(mod, variables = list(hp = c(90, 100)))
avg_comparisons(mod, variables = list(hp = "iqr"))
avg_comparisons(mod, variables = list(hp = "sd"))
avg_comparisons(mod, variables = list(hp = "minmax"))
# using a function to specify a custom difference in one regressor
dat <- mtcars
dat$new_hp <- 49 * (dat$hp - min(dat$hp)) / (max(dat$hp) - min(dat$hp)) + 1
modlog <- lm(mpg ~ log(new_hp) + factor(cyl), data = dat)
fdiff <- function(x) data.frame(x, x + 10)
avg_comparisons(modlog, variables = list(new_hp = fdiff))
# Adjusted Risk Ratio
mod <- glm(vs ~ mpg, data = mtcars, family = binomial)
avg_comparisons(mod, comparison = "lnratioavg", transform = exp)
# Adjusted Risk Ratio: Manual specification of the `comparison`
avg_comparisons(
mod,
comparison = function(hi, lo) log(mean(hi) / mean(lo)),
transform = exp)
# cross contrasts
mod <- lm(mpg ~ factor(cyl) * factor(gear) + hp, data = mtcars)
avg_comparisons(mod, variables = c("cyl", "gear"), cross = TRUE)
# variable-specific contrasts
avg_comparisons(mod, variables = list(gear = "sequential", hp = 10))
# hypothesis test: is the `hp` marginal effect at the mean equal to the `drat` marginal effect
mod <- lm(mpg ~ wt + drat, data = mtcars)
comparisons(
mod,
newdata = "mean",
hypothesis = "wt = drat")
# same hypothesis test using row indices
comparisons(
mod,
newdata = "mean",
hypothesis = "b1 - b2 = 0")
# same hypothesis test using numeric vector of weights
comparisons(
mod,
newdata = "mean",
hypothesis = c(1, -1))
# two custom contrasts using a matrix of weights
lc <- matrix(
c(
1, -1,
2, 3),
ncol = 2)
comparisons(
mod,
newdata = "mean",
hypothesis = lc)
# Effect of a 1 group-wise standard deviation change
# First we calculate the SD in each group of `cyl`
# Second, we use that SD as the treatment size in the `variables` argument
library(dplyr)
mod <- lm(mpg ~ hp + factor(cyl), mtcars)
tmp <- mtcars %>%
group_by(cyl) %>%
mutate(hp_sd = sd(hp))
avg_comparisons(mod,
variables = list(hp = function(x) data.frame(x, x + tmp$hp_sd)),
by = "cyl")
# `by` argument
mod <- lm(mpg ~ hp * am * vs, data = mtcars)
comparisons(mod, by = TRUE)
mod <- lm(mpg ~ hp * am * vs, data = mtcars)
avg_comparisons(mod, variables = "hp", by = c("vs", "am"))
library(nnet)
mod <- multinom(factor(gear) ~ mpg + am * vs, data = mtcars, trace = FALSE)
by <- data.frame(
group = c("3", "4", "5"),
by = c("3,4", "3,4", "5"))
comparisons(mod, type = "probs", by = by)
```
---
## Slopes (aka Partial derivatives, Marginal Effects, or Trends) {.unnumbered}
### Description
Partial derivative of the regression equation with respect to a regressor of interest.
-
slopes(): unit-level (conditional) estimates.
-
avg_slopes(): average (marginal) estimates.
The newdata argument and the datagrid() function can be used to control where statistics are evaluated in the predictor space: "at observed values", "at the mean", "at representative values", etc.
See the slopes vignette and package website for worked examples and case studies:
Warning: Slopes and elasticities can only be calculated for continuous numeric variables. The slopes() functions will automatically revert to comparisons() for binary or categorical variables.
### Usage
slopes(
model,
newdata = NULL,
variables = NULL,
type = NULL,
by = FALSE,
vcov = TRUE,
conf_level = 0.95,
slope = "dydx",
wts = FALSE,
hypothesis = NULL,
equivalence = NULL,
df = Inf,
eps = NULL,
numderiv = "fdforward",
...
)
avg_slopes(
model,
newdata = NULL,
variables = NULL,
type = NULL,
by = TRUE,
vcov = TRUE,
conf_level = 0.95,
slope = "dydx",
wts = FALSE,
hypothesis = NULL,
equivalence = NULL,
df = Inf,
eps = NULL,
numderiv = "fdforward",
...
)
### Arguments
model |
Model object
|
newdata |
Grid of predictor values at which we evaluate the slopes.
- Warning: Please avoid modifying your dataset between fitting the model and calling a
marginaleffects function. This can sometimes lead to unexpected results.
-
NULL (default): Unit-level slopes for each observed value in the dataset (empirical distribution). The dataset is retrieved using insight::get_data(), which tries to extract data from the environment. This may produce unexpected results if the original data frame has been altered since fitting the model.
-
datagrid() call to specify a custom grid of regressors. For example:
-
newdata = datagrid(cyl = c(4, 6)): cyl variable equal to 4 and 6 and other regressors fixed at their means or modes.
- See the Examples section and the
datagrid() documentation.
-
subset() call with a single argument to select a subset of the dataset used to fit the model, ex: newdata = subset(treatment == 1)
-
dplyr::filter() call with a single argument to select a subset of the dataset used to fit the model, ex: newdata = filter(treatment == 1)
- string:
- "mean": Slopes evaluated when each predictor is held at its mean or mode.
- "median": Slopes evaluated when each predictor is held at its median or mode.
- "balanced": Slopes evaluated on a balanced grid with every combination of categories and numeric variables held at their means.
- "tukey": Slopes evaluated at Tukey's 5 numbers.
- "grid": Slopes evaluated on a grid of representative numbers (Tukey's 5 numbers and unique values of categorical predictors).
|
variables |
Focal variables
-
NULL: compute slopes or comparisons for all the variables in the model object (can be slow).
- Character vector: subset of variables (usually faster).
|
type |
string indicates the type (scale) of the predictions used to
compute contrasts or slopes. This can differ based on the model
type, but will typically be a string such as: "response", "link", "probs",
or "zero". When an unsupported string is entered, the model-specific list of
acceptable values is returned in an error message. When type is NULL, the
first entry in the error message is used by default. See the Type section in the documentation below.
|
by |
Aggregate unit-level estimates (aka, marginalize, average over). Valid inputs:
-
FALSE: return the original unit-level estimates.
-
TRUE: aggregate estimates for each term.
- Character vector of column names in
newdata or in the data frame produced by calling the function without the by argument.
- Data frame with a
by column of group labels, and merging columns shared by newdata or the data frame produced by calling the same function without the by argument.
- See examples below.
- For more complex aggregations, you can use the
FUN argument of the hypotheses() function. See that function's documentation and the Hypothesis Test vignettes on the marginaleffects website.
|
vcov |
Type of uncertainty estimates to report (e.g., for robust standard errors). Acceptable values:
- FALSE: Do not compute standard errors. This can speed up computation considerably.
- TRUE: Unit-level standard errors using the default
vcov(model) variance-covariance matrix.
- String which indicates the kind of uncertainty estimates to return.
- Heteroskedasticity-consistent:
"HC", "HC0", "HC1", "HC2", "HC3", "HC4", "HC4m", "HC5". See ?sandwich::vcovHC
- Heteroskedasticity and autocorrelation consistent:
"HAC"
- Mixed-Models degrees of freedom: "satterthwaite", "kenward-roger"
- Other:
"NeweyWest", "KernHAC", "OPG". See the sandwich package documentation.
- "rsample", "boot", "fwb", and "simulation" are passed to the
method argument of the inferences() function. To customize the bootstrap or simulation process, call inferences() directly.
- One-sided formula which indicates the name of cluster variables (e.g.,
~unit_id). This formula is passed to the cluster argument of the sandwich::vcovCL function.
- Square covariance matrix
- Function which returns a covariance matrix (e.g.,
stats::vcov(model))
|
conf_level |
numeric value between 0 and 1. Confidence level to use to build a confidence interval.
|
slope |
string indicates the type of slope or (semi-)elasticity to compute:
- "dydx": dY/dX
- "eyex": dY/dX * Y / X
- "eydx": dY/dX * Y
- "dyex": dY/dX / X
- Y is the predicted value of the outcome; X is the observed value of the predictor.
|
wts |
logical, string or numeric: weights to use when computing average predictions, contrasts or slopes. These weights only affect the averaging in avg_*() or with the by argument, and not unit-level estimates. See ?weighted.mean
- string: column name of the weights variable in
newdata. When supplying a column name to wts, it is recommended to supply the original data (including the weights variable) explicitly to newdata.
- numeric: vector of length equal to the number of rows in the original data or in
newdata (if supplied).
- FALSE: Equal weights.
- TRUE: Extract weights from the fitted object with
insight::find_weights() and use them when taking weighted averages of estimates. Warning: newdata=datagrid() returns a single average weight, which is equivalent to using wts=FALSE
|
hypothesis |
specify a hypothesis test or custom contrast using a number , formula, string equation, vector, matrix, or function.
- Number: The null hypothesis used in the computation of Z and p (before applying
transform).
- String: Equation to specify linear or non-linear hypothesis tests. Two-tailed tests must include an equal
= sign. One-tailed tests must start with < or >. If the terms in coef(object) uniquely identify estimates, they can be used in the formula. Otherwise, use b1, b2, etc. to identify the position of each parameter. The b* wildcard can be used to test hypotheses on all estimates. When the hypothesis string represents a two-sided equation, the estimate column holds the value of the left side minus the right side of the equation. If a named vector is used, the names are used as labels in the output. Examples:
-
hp = drat
-
hp + drat = 12
-
b1 + b2 + b3 = 0
-
b* / b1 = 1
-
<= 0
-
>= -3.5
-
b1 >= 10
- Formula:
lhs ~ rhs | group
-
lhs
-
ratio (null = 1)
-
difference (null = 0)
- Leave empty for default value
-
rhs
-
pairwise and revpairwise: pairwise differences between estimates in each row.
-
reference: differences between the estimates in each row and the estimate in the first row.
-
sequential: difference between an estimate and the estimate in the next row.
-
meandev: difference between an estimate and the mean of all estimates.
-
meanotherdev: difference between an estimate and the mean of all other estimates, excluding the current one.
-
poly: polynomial contrasts, as computed by the stats::contr.poly() function.
-
helmert: Helmert contrasts, as computed by the stats::contr.helmert() function. Contrast 2nd level to the first, 3rd to the average of the first two, and so on.
-
trt_vs_ctrl: difference between the mean of estimates (except the first) and the first estimate.
-
I(fun(x)): custom function to manipulate the vector of estimates x. The function fun() can return multiple (potentially named) estimates.
-
group (optional)
- Column name of
newdata. Conduct hypothesis tests withing subsets of the data.
- Examples:
-
~ poly
-
~ sequential | groupid
-
~ reference
-
ratio ~ pairwise
-
difference ~ pairwise | groupid
-
~ I(x - mean(x)) | groupid
-
~ I(\(x) c(a = x[1], b = mean(x[2:3]))) | groupid
- Matrix or Vector: Each column is a vector of weights. The the output is the dot product between these vectors of weights and the vector of estimates. The matrix can have column names to label the estimates.
- Function:
- Accepts an argument
x: object produced by a marginaleffects function or a data frame with column rowid and estimate
- Returns a data frame with columns
term and estimate (mandatory) and rowid (optional).
- The function can also accept optional input arguments:
newdata, by, draws.
- This function approach will not work for Bayesian models or with bootstrapping. In those cases, it is easy to use
get_draws() to extract and manipulate the draws directly.
- See the Examples section below and the vignette: https://marginaleffects.com/chapters/hypothesis.html
- Warning: When calling
predictions() with type="invlink(link)" (the default in some models), hypothesis is tested and p values are computed on the link scale.
|
equivalence |
Numeric vector of length 2: bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. For bayesian models, this report the proportion of posterior draws in the interval and the ROPE. See Details section below.
|
df |
Degrees of freedom used to compute p values and confidence intervals.
- A single numeric value between 1 and
Inf, or a numeric vector with length equal to the number of rows in the output. When df is Inf, the normal distribution is used. When df is finite, the t distribution is used.
- "residual": Calls insight::get_df to extract degrees of freedom from the model automatically.
- "satterthwaite" or "kenward-roger": Use the Satterthwaite or Kenward-Roger approximation to compute degrees of freedom in mixed effects models.
|
eps |
NULL or numeric value which determines the step size to use when
calculating numerical derivatives: (f(x+eps)-f(x))/eps. When eps is
NULL, the step size is 0.0001 multiplied by the difference between
the maximum and minimum values of the variable with respect to which we
are taking the derivative. Changing eps may be necessary to avoid
numerical problems in certain models.
|
numderiv |
string or list of strings indicating the method to use to for the numeric differentiation used in to compute delta method standard errors.
- "fdforward": finite difference method with forward differences (default)
- "fdcenter": finite difference method with central differences
- "richardson": Richardson extrapolation method
- Extra arguments can be specified by passing a list to the
numDeriv argument, with the name of the method first and named arguments following, ex: numderiv=list("fdcenter", eps = 1e-5). When an unknown argument is used, marginaleffects prints the list of valid arguments for each method.
|
... |
Additional arguments are passed to the predict() method
supplied by the modeling package.These arguments are particularly useful
for mixed-effects or bayesian models (see the online vignettes on the
marginaleffects website). Available arguments can vary from model to
model, depending on the range of supported arguments by each modeling
package. See the "Model-Specific Arguments" section of the
?slopes documentation for a non-exhaustive list of available
arguments.
|
### Details
A "slope" or "marginal effect" is the partial derivative of the regression equation
with respect to a variable in the model. This function uses automatic
differentiation to compute slopes for a vast array of models,
including non-linear models with transformations (e.g., polynomials).
Uncertainty estimates are computed using the delta method.
Numerical derivatives for the slopes function are calculated
using a simple epsilon difference approach: $\partial Y / \partial X = (f(X + \varepsilon/2) - f(X-\varepsilon/2)) / \varepsilon$,
where f is the predict() method associated with the model class, and
$\varepsilon$ is determined by the eps argument.
### Value
A data.frame with one row per estimate. This data frame is pretty-printed by default, but users can interact with it as a regular data frame, with functions like nrow(), head(), colnames(), etc. Values can be extracted using standard [,] or \$ operators, and manipulated using external packages like dplyr or data.table.
Columns may include:
-
rowid: row number of the newdata data frame
-
group: (optional) value of the grouped outcome (e.g., categorical outcome models)
-
term: the focal variable.
-
estimate: an estimate of the prediction, counterfactual comparison, or slope.
-
std.error: standard errors computed via the delta method.
-
p.value: p value associated to the estimate column. The null is determined by the hypothesis argument (0 by default).
-
s.value: Shannon information transforms of p values. See the S values vignette at https://marginaleffects.com the marginaleffects website.
-
conf.low: lower bound of the confidence (or credible) interval defined by the conf_level argument.
-
conf.high: upper bound of the confidence (or credible) interval defined by the conf_level argument.
-
predicted_lo: predicted outcome for the "low" value of the focal predictor in a counterfactual comparison.
-
predicted_hi: predicted outcome for the "high" value of the focal predictor in a counterfactual comparison.
-
p.rope.unconditional: share of posterior draws in the interval specified by the equivalence argument. This is only available for Bayesian models.
-
p.rope.conditional: share of posterior draws in the interval specified by the equivalence argument, among draws in the confidence interval. This is only available for Bayesian models.
-
rope: share of the posterior draws between conf.low and conf.high that are covered by the interval specified by the equivalence argument.
-
statistic.noninf: test statistic for non-inferiority test (when equivalence argument is used).
-
statistic.nonsup: test statistic for non-superiority test (when equivalence argument is used).
-
p.value.noninf: p-value for non-inferiority test (when equivalence argument is used).
-
p.value.nonsup: p-value for non-superiority test (when equivalence argument is used).
-
p.value.equiv: p-value for equivalence test using Two One-Sided Tests (TOST) approach (when equivalence argument is used).
See ?print.marginaleffects for printing options.
The data.frames produced by marginaleffects stores an attribute that holds many internal objects, such as the original model, data, and much other information that can be used for post-processing. This information can be inspected using the components() function.
Warning: The internal attributes retrieved by components() are not considered part of the public API of the package. Their names and contents can change without warning or notice. Users should not rely on them.
Warning: In some cases, the internal attributes used by marginaleffects() can use up a substantial amount of memory. To clear this data, use the prune() function or set options(marginaleffects_lean=TRUE).
### Functions
-
avg_slopes(): Average slopes
### Standard errors using the delta method
Standard errors for all quantities estimated by marginaleffects can be obtained via the delta method. This requires differentiating a function with respect to the coefficients in the model using a finite difference approach. In some models, the delta method standard errors can be sensitive to various aspects of the numeric differentiation strategy, including the step size. By default, the step size is set to 1e-8, or to 1e-4 times the smallest absolute model coefficient, whichever is largest.
marginaleffects can delegate numeric differentiation to the numDeriv package, which allows more flexibility. To do this, users can pass arguments to the numDeriv::jacobian function through a global option. For example:
-
options(marginaleffects_numDeriv = list(method = "simple", method.args = list(eps = 1e-6)))
-
options(marginaleffects_numDeriv = list(method = "Richardson", method.args = list(eps = 1e-5)))
-
options(marginaleffects_numDeriv = NULL)
See the "Uncertainty" chapter on the marginaleffects website for more details on the computation of standard errors, bootstrapping, and more:
https://marginaleffects.com/chapters/uncertainty.html
### Model-Specific Arguments
Some model types allow model-specific arguments to modify the nature of
marginal effects, predictions, marginal means, and contrasts. Please report
other package-specific predict() arguments on Github so we can add them to
the table below.
https://github.com/vincentarelbundock/marginaleffects/issues
|
Package | Class | Argument | Documentation |
brms | brmsfit | ndraws | brms::posterior_predict |
|
| | re_formula | brms::posterior_predict |
lme4 | merMod | re.form | lme4::predict.merMod |
|
| | allow.new.levels | lme4::predict.merMod |
glmmTMB | glmmTMB | re.form | glmmTMB::predict.glmmTMB |
|
| | allow.new.levels | glmmTMB::predict.glmmTMB |
|
| | zitype | glmmTMB::predict.glmmTMB |
mgcv | bam | exclude | mgcv::predict.bam |
|
| gam | exclude | mgcv::predict.gam |
robustlmm | rlmerMod | re.form | robustlmm::predict.rlmerMod |
|
| | allow.new.levels | robustlmm::predict.rlmerMod |
MCMCglmm | MCMCglmm | ndraws | |
sampleSelection | selection | part | sampleSelection::predict.selection |
|
|
### Bayesian posterior summaries
By default, credible intervals in bayesian models are built as equal-tailed
intervals. This can be changed to a highest density interval by setting a global
option:
options("marginaleffects_posterior_interval" = "eti")
options("marginaleffects_posterior_interval" = "hdi")
By default, the center of the posterior distribution in bayesian models is
identified by the median. Users can use a different summary function by setting a
global option:
options("marginaleffects_posterior_center" = "mean")
options("marginaleffects_posterior_center" = "median")
When estimates are averaged using the by argument, the tidy() function, or
the summary() function, the posterior distribution is marginalized twice over.
First, we take the average across units but within each iteration of the
MCMC chain, according to what the user requested in by argument or
tidy()/summary() functions. Then, we identify the center of the resulting
posterior using the function supplied to the
"marginaleffects_posterior_center" option (the median by default).
### Equivalence, Inferiority, Superiority
$\theta$ is an estimate, $\sigma_\theta$ its estimated standard error, and $[a, b]$ are the bounds of the interval supplied to the equivalence argument.
Non-inferiority:
- $H_0$: $\theta \leq a$
- $H_1$: $\theta > a$
- $t=(\theta - a)/\sigma_\theta$
- p: Upper-tail probability
Non-superiority:
- $H_0$: $\theta \geq b$
- $H_1$: $\theta < b$
- $t=(\theta - b)/\sigma_\theta$
- p: Lower-tail probability
Equivalence: Two One-Sided Tests (TOST)
- p: Maximum of the non-inferiority and non-superiority p values.
Thanks to Russell V. Lenth for the excellent emmeans package and documentation which inspired this feature.
### Parallel computation
The slopes() and comparisons() functions can use parallelism to
speed up computation. Operations are parallelized for the computation of
standard errors, at the model coefficient level. There is always
considerable overhead when using parallel computation, mainly involved
in passing the whole dataset to the different processes. Thus, parallel
computation is most likely to be useful when the model includes many parameters
and the dataset is relatively small.
Warning: In many cases, parallel processing will not be useful at all.
To activate parallel computation, users must load the future.apply package,
call plan() function, and set a global option.
options(marginaleffects_parallel = TRUE): parallelize delta method computation of standard errors.
options(marginaleffects_parallel_inferences = TRUE): parallelize "rsample" or "fwb" bootstrap computation in inferences().
options(marginaleffects_parallel_packages = TRUE): vector of strings with the names of modeling packages used to fit the model, ex: c("survival", "splines")
For example:
library(future.apply)
plan("multisession", workers = 4)
options(marginaleffects_parallel = FALSE)
options(marginaleffects_parallel_inferences = TRUE)
options(marginaleffects_parallel_packages = c("survival", "splines"))
slopes(model)
marginaleffects altogether, you can set a global option:
options(marginaleffects_parallel = FALSE)
marginaleffects functions are evaluated in this order:
-
newdata
-
variables
-
comparison and slope
-
by
-
vcov
-
hypothesis
-
transform
### Global options
The behavior of marginaleffects functions can be modified by setting global options.
Disable some safety checks and warnings:
-
options(marginaleffects_startup_message = FALSE)
- Disable the startup message printed on
library(marginaleffects).
-
options(marginaleffects_safe = FALSE)
- Disable safety checks and warnings.
-
options(marginaleffects_print_omit = c("p.value", "s.value"))
- Omit some columns from the printed output.
Enforce lean return objects, sans information about the original model and
data, and other ancillary attributes. Note that this will disable some
advanced post-processing features and functions like hypotheses.
options(marginaleffects_lean = TRUE)
-
marginaleffects_plot_gray: logical. If TRUE, the default color of the plot is gray. Default is FALSE.
### Types
The type argument determines the scale of the predictions used to compute quantities of interest with functions from the marginaleffects package. Admissible values for type depend on the model object. When users specify an incorrect value for type, marginaleffects will raise an informative error with a list of valid type values for the specific model object. The first entry in the list in that error message is the default type.
The invlink(link) is a special type defined by marginaleffects. It is available for some (but not all) models, and only for the predictions() function. With this link type, we first compute predictions on the link scale, then we use the inverse link function to backtransform the predictions to the response scale. This is useful for models with non-linear link functions as it can ensure that confidence intervals stay within desirable bounds, ex: 0 to 1 for a logit model. Note that an average of estimates with type="invlink(link)" will not always be equivalent to the average of estimates with type="response". This type is default when calling predictions(). It is available—but not default—when calling avg_predictions() or predictions() with the by argument.
Some of the most common type values are:
|
class | type |
|
Gam | invlink(link), response, link |
|
Gls | lp |
|
MCMCglmm | response |
|
bam | response, link |
|
bart | ev, ppd |
|
betareg | response, link, precision, quantile, variance |
|
bife | response, link |
|
bracl | probs |
|
brglmFit | response, link |
|
brmsfit | response, link, prediction, average |
|
brmultinom | probs, class |
|
clm | prob, cum.prob, linear.predictor |
|
clogit | expected, lp, risk, survival |
|
coxph | survival, expected, lp, risk |
|
coxph_weightit | survival, expected, lp, risk |
|
crch | response, location, scale, density |
|
fixest | invlink(link), response, link |
|
flexsurvreg | survival, response, mean, link, lp, linear, rmst, hazard, cumhaz |
|
gam | response, link |
|
geeglm | response, link |
|
glimML | response, link |
|
glm | invlink(link), response, link |
|
glm_weightit | invlink(link), probs, response, lp, link |
|
glmerMod | response, link |
|
glmgee | response |
|
glmmPQL | response, link |
|
glmmTMB | response, link, conditional, zprob, zlink, disp |
|
glmrob | response, link |
|
glmx | response |
|
hetprob | pr, xb |
|
hurdle | response, prob, count, zero |
|
hxlr | location, cumprob, scale, density |
|
iv_robust | response |
|
ivpml | pr, xb |
|
ivreg | response |
|
lda | class, posterior |
|
lm | response |
|
lm_robust | response |
|
lmerMod | response |
|
lmerModLmerTest | response |
|
lmrob | response |
|
lrm | fitted, lp, mean |
|
mblogit | response, latent, link |
|
mclogit | response, latent, link |
|
mhurdle | E, Ep, p |
|
model_fit | numeric, prob, class |
|
multinom | probs, latent |
|
multinom_weightit | probs, response, mean |
|
mvgam | response, link, expected, detection, latent_N |
|
negbin | invlink(link), response, link |
|
ols | lp |
|
oohbchoice | probability, utility |
|
ordinal_weightit | probs, response, link, lp, mean |
|
orm | fitted, mean, lp |
|
polr | probs |
|
rendo.base | response, link |
|
rlm | response |
|
selection | response, link, unconditional, conditional |
|
speedglm | response, link |
|
speedlm | response |
|
stanreg | response, link |
|
survreg | response, link, quantile |
|
svyglm | response, link |
|
svyolr | probs |
|
tobit | response, link |
|
tobit1 | expvalue, linpred, prob |
|
workflow | numeric, prob, class |
|
zeroinfl | response, prob, count, zero |
|
|
### References
- Arel-Bundock V, Greifer N, Heiss A (2024). “How to Interpret Statistical Models Using marginaleffects for R and Python.” Journal of Statistical Software, 111(9), 1-32. doi:10.18637/jss.v111.i09 [doi:10.18637/jss.v111.i09](https://doi.org/10.18637/jss.v111.i09)
- Arel-Bundock (2026). "Model to Meaning: How to interpret statistical models in R and Python." CRC Press. https://routledge.com/9781032908724
- Greenland S. 2019. "Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values." The American Statistician. 73(S1): 106–114.
- Cole, Stephen R, Jessie K Edwards, and Sander Greenland. 2020. "Surprise!" American Journal of Epidemiology 190 (2): 191–93. [doi:10.1093/aje/kwaa136](https://doi.org/10.1093/aje/kwaa136)
### Examples
```{r, warning=FALSE, message=FALSE, eval=FALSE}
library("marginaleffects")
library("marginaleffects")
# Unit-level (conditional) Marginal Effects
mod <- glm(am ~ hp * wt, data = mtcars, family = binomial)
mfx <- slopes(mod)
head(mfx)
# Average Marginal Effect (AME)
avg_slopes(mod, by = TRUE)
# Marginal Effect at the Mean (MEM)
slopes(mod, newdata = datagrid())
# Marginal Effect at User-Specified Values
# Variables not explicitly included in `datagrid()` are held at their means
slopes(mod, newdata = datagrid(hp = c(100, 110)))
# Group-Average Marginal Effects (G-AME)
# Calculate marginal effects for each observation, and then take the average
# marginal effect within each subset of observations with different observed
# values for the `cyl` variable:
mod2 <- lm(mpg ~ hp * cyl, data = mtcars)
avg_slopes(mod2, variables = "hp", by = "cyl")
# Marginal Effects at User-Specified Values (counterfactual)
# Variables not explicitly included in `datagrid()` are held at their
# original values, and the whole dataset is duplicated once for each
# combination of the values in `datagrid()`
mfx <- slopes(mod,
newdata = datagrid(
hp = c(100, 110),
grid_type = "counterfactual"))
head(mfx)
# Heteroskedasticity robust standard errors
mfx <- slopes(mod, vcov = sandwich::vcovHC(mod))
head(mfx)
# hypothesis test: is the `hp` marginal effect at the mean equal to the `drat` marginal effect
mod <- lm(mpg ~ wt + drat, data = mtcars)
slopes(
mod,
newdata = "mean",
hypothesis = "wt = drat")
# same hypothesis test using row indices
slopes(
mod,
newdata = "mean",
hypothesis = "b1 - b2 = 0")
# same hypothesis test using numeric vector of weights
slopes(
mod,
newdata = "mean",
hypothesis = c(1, -1))
# two custom contrasts using a matrix of weights
lc <- matrix(
c(
1, -1,
2, 3),
ncol = 2)
colnames(lc) <- c("Contrast A", "Contrast B")
slopes(
mod,
newdata = "mean",
hypothesis = lc)
```
---
## (Non-)Linear Tests for Null Hypotheses, Joint Hypotheses, Equivalence, Non Superiority, and Non Inferiority {.unnumbered}
### Description
Uncertainty estimates are calculated as first-order approximate standard errors for linear or non-linear functions of a vector of random variables with known or estimated covariance matrix. In that sense, hypotheses emulates the behavior of the excellent and well-established car::deltaMethod and car::linearHypothesis functions, but it supports more models; requires fewer dependencies; expands the range of tests to equivalence and superiority/inferiority; and offers convenience features like robust standard errors.
To learn more, read the hypothesis tests vignette, visit the
package website:
Warning #1: Tests are conducted directly on the scale defined by the type argument. For some models, it can make sense to conduct hypothesis or equivalence tests on the "link" scale instead of the "response" scale which is often the default.
Warning #2: For hypothesis tests on objects produced by the marginaleffects package, it is safer to use the hypothesis argument of the original function. Using hypotheses() may not work in certain environments, in lists, or when working programmatically with *apply style functions.
Warning #3: The tests assume that the hypothesis expression is (approximately) normally distributed, which for non-linear functions of the parameters may not be realistic. More reliable confidence intervals can be obtained using the inferences() function with method = "boot".
### Usage
hypotheses(
model = NULL,
hypothesis = NULL,
vcov = TRUE,
conf_level = NULL,
df = NULL,
equivalence = NULL,
joint = FALSE,
joint_test = "f",
multcomp = FALSE,
numderiv = "fdforward",
...
)
### Arguments
model |
Model object or object generated by the comparisons(), slopes(), or predictions() functions.
|
hypothesis |
specify a hypothesis test or custom contrast using a number , formula, string equation, vector, matrix, or function.
- Number: The null hypothesis used in the computation of Z and p (before applying
transform).
- String: Equation to specify linear or non-linear hypothesis tests. Two-tailed tests must include an equal
= sign. One-tailed tests must start with < or >. If the terms in coef(object) uniquely identify estimates, they can be used in the formula. Otherwise, use b1, b2, etc. to identify the position of each parameter. The b* wildcard can be used to test hypotheses on all estimates. When the hypothesis string represents a two-sided equation, the estimate column holds the value of the left side minus the right side of the equation. If a named vector is used, the names are used as labels in the output. Examples:
-
hp = drat
-
hp + drat = 12
-
b1 + b2 + b3 = 0
-
b* / b1 = 1
-
<= 0
-
>= -3.5
-
b1 >= 10
- Formula:
lhs ~ rhs | group
-
lhs
-
ratio (null = 1)
-
difference (null = 0)
- Leave empty for default value
-
rhs
-
pairwise and revpairwise: pairwise differences between estimates in each row.
-
reference: differences between the estimates in each row and the estimate in the first row.
-
sequential: difference between an estimate and the estimate in the next row.
-
meandev: difference between an estimate and the mean of all estimates.
-
meanotherdev: difference between an estimate and the mean of all other estimates, excluding the current one.
-
poly: polynomial contrasts, as computed by the stats::contr.poly() function.
-
helmert: Helmert contrasts, as computed by the stats::contr.helmert() function. Contrast 2nd level to the first, 3rd to the average of the first two, and so on.
-
trt_vs_ctrl: difference between the mean of estimates (except the first) and the first estimate.
-
I(fun(x)): custom function to manipulate the vector of estimates x. The function fun() can return multiple (potentially named) estimates.
-
group (optional)
- Column name of
newdata. Conduct hypothesis tests withing subsets of the data.
- Examples:
-
~ poly
-
~ sequential | groupid
-
~ reference
-
ratio ~ pairwise
-
difference ~ pairwise | groupid
-
~ I(x - mean(x)) | groupid
-
~ I(\(x) c(a = x[1], b = mean(x[2:3]))) | groupid
- Matrix or Vector: Each column is a vector of weights. The the output is the dot product between these vectors of weights and the vector of estimates. The matrix can have column names to label the estimates.
- Function:
- Accepts an argument
x: object produced by a marginaleffects function or a data frame with column rowid and estimate
- Returns a data frame with columns
term and estimate (mandatory) and rowid (optional).
- The function can also accept optional input arguments:
newdata, by, draws.
- This function approach will not work for Bayesian models or with bootstrapping. In those cases, it is easy to use
get_draws() to extract and manipulate the draws directly.
- See the Examples section below and the vignette: https://marginaleffects.com/chapters/hypothesis.html
- Warning: When calling
predictions() with type="invlink(link)" (the default in some models), hypothesis is tested and p values are computed on the link scale.
|
vcov |
Type of uncertainty estimates to report (e.g., for robust standard errors). Acceptable values:
- FALSE: Do not compute standard errors. This can speed up computation considerably.
- TRUE: Unit-level standard errors using the default
vcov(model) variance-covariance matrix.
- String which indicates the kind of uncertainty estimates to return.
- Heteroskedasticity-consistent:
"HC", "HC0", "HC1", "HC2", "HC3", "HC4", "HC4m", "HC5". See ?sandwich::vcovHC
- Heteroskedasticity and autocorrelation consistent:
"HAC"
- Mixed-Models degrees of freedom: "satterthwaite", "kenward-roger"
- Other:
"NeweyWest", "KernHAC", "OPG". See the sandwich package documentation.
- "rsample", "boot", "fwb", and "simulation" are passed to the
method argument of the inferences() function. To customize the bootstrap or simulation process, call inferences() directly.
- One-sided formula which indicates the name of cluster variables (e.g.,
~unit_id). This formula is passed to the cluster argument of the sandwich::vcovCL function.
- Square covariance matrix
- Function which returns a covariance matrix (e.g.,
stats::vcov(model))
|
conf_level |
NULL or numeric value between 0 and 1. Confidence level to use to build a confidence interval. When NULL and model was generated by marginaleffects, the confidence level is taken from the conf_level attribute of the model. Otherwise, the default value is 0.95.
|
df |
Degrees of freedom used to compute p values and confidence intervals.
- A single numeric value between 1 and
Inf, or a numeric vector with length equal to the number of rows in the output. When df is Inf, the normal distribution is used. When df is finite, the t distribution is used.
- "residual": Calls insight::get_df to extract degrees of freedom from the model automatically.
- When using
joint_test="f", the df argument should be a numeric vector of length 2.
|
equivalence |
Numeric vector of length 2: bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. For bayesian models, this report the proportion of posterior draws in the interval and the ROPE. See Details section below.
|
joint |
Joint test of statistical significance. The null hypothesis value can be set using the hypothesis argument.
- FALSE: Hypotheses are not tested jointly.
- TRUE: All parameters are tested jointly.
- String: A regular expression to match parameters to be tested jointly.
grep(joint, perl = TRUE)
- Character vector of parameter names to be tested. Characters refer to the names of the vector returned by
marginaleffects::get_coef(object).
- Integer vector of indices. Which parameters positions to test jointly.
- Note: When using the
joint argument, the hypothesis argument is limited to NULL or numeric values. Users can chain multiple hypotheses() for complex joint hypothesis tests.
|
joint_test |
A character string specifying the type of test, either "f" or "chisq". The null hypothesis is set by the hypothesis argument, with default null equal to 0 for all parameters.
|
multcomp |
Logical or string. If TRUE or string, apply multiple comparison adjustment to the p values and report family-wise confidence intervals. Valid strings: "holm", "hochberg", "hommel", "bonferroni", "BH", "BY", "fdr", "single-step", "Shaffer", "Westfall", "free". When multcomp is TRUE, the "holm" method is used.
|
numderiv |
string or list of strings indicating the method to use to for the numeric differentiation used in to compute delta method standard errors.
- "fdforward": finite difference method with forward differences (default)
- "fdcenter": finite difference method with central differences
- "richardson": Richardson extrapolation method
- Extra arguments can be specified by passing a list to the
numDeriv argument, with the name of the method first and named arguments following, ex: numderiv=list("fdcenter", eps = 1e-5). When an unknown argument is used, marginaleffects prints the list of valid arguments for each method.
|
... |
Additional arguments are passed to the predict() method
supplied by the modeling package.These arguments are particularly useful
for mixed-effects or bayesian models (see the online vignettes on the
marginaleffects website). Available arguments can vary from model to
model, depending on the range of supported arguments by each modeling
package. See the "Model-Specific Arguments" section of the
?slopes documentation for a non-exhaustive list of available
arguments.
|
### Joint hypothesis tests
The test statistic for the joint Wald test is calculated as (R * theta_hat - r)' * inv(R * V_hat * R') * (R * theta_hat - r) / Q,
where theta_hat is the vector of estimated parameters, V_hat is the estimated covariance matrix, R is a Q x P matrix for testing Q hypotheses on P parameters,
r is a Q x 1 vector for the null hypothesis, and Q is the number of rows in R. If the test is a Chi-squared test, the test statistic is not normalized.
The p-value is then calculated based on either the F-distribution (for F-test) or the Chi-squared distribution (for Chi-squared test).
For the F-test, the degrees of freedom are Q and (n - P), where n is the sample size and P is the number of parameters.
For the Chi-squared test, the degrees of freedom are Q.
### Equivalence, Inferiority, Superiority
$\theta$ is an estimate, $\sigma_\theta$ its estimated standard error, and $[a, b]$ are the bounds of the interval supplied to the equivalence argument.
Non-inferiority:
- $H_0$: $\theta \leq a$
- $H_1$: $\theta > a$
- $t=(\theta - a)/\sigma_\theta$
- p: Upper-tail probability
Non-superiority:
- $H_0$: $\theta \geq b$
- $H_1$: $\theta < b$
- $t=(\theta - b)/\sigma_\theta$
- p: Lower-tail probability
Equivalence: Two One-Sided Tests (TOST)
- p: Maximum of the non-inferiority and non-superiority p values.
Thanks to Russell V. Lenth for the excellent emmeans package and documentation which inspired this feature.
### Examples
```{r, warning=FALSE, message=FALSE, eval=TRUE}
library("marginaleffects")
mod <- lm(mpg ~ hp + wt + factor(cyl), data = mtcars)
hypotheses(mod)
# Test of equality between coefficients
hypotheses(mod, hypothesis = "hp = wt")
# Non-linear function
hypotheses(mod, hypothesis = "exp(hp + wt) = 0.1")
# Robust standard errors
hypotheses(mod, hypothesis = "hp = wt", vcov = "HC3")
# b1, b2, ... shortcuts can be used to identify the position of the
# parameters of interest in the output of
hypotheses(mod, hypothesis = "b2 = b3")
# wildcard
hypotheses(mod, hypothesis = "b* / b2 = 1")
# term names with special characters have to be enclosed in backticks
hypotheses(mod, hypothesis = "`factor(cyl)6` = `factor(cyl)8`")
mod2 <- lm(mpg ~ hp * drat, data = mtcars)
hypotheses(mod2, hypothesis = "`hp:drat` = drat")
# predictions(), comparisons(), and slopes()
mod <- glm(am ~ hp + mpg, data = mtcars, family = binomial)
cmp <- comparisons(mod, newdata = "mean")
hypotheses(cmp, hypothesis = "b1 = b2")
mfx <- slopes(mod, newdata = "mean")
hypotheses(cmp, hypothesis = "b2 = 0.2")
pre <- predictions(mod, newdata = datagrid(hp = 110, mpg = c(30, 35)))
hypotheses(pre, hypothesis = "b1 = b2")
# The `hypothesis` argument can be used to compute standard errors for fitted values
mod <- glm(am ~ hp + mpg, data = mtcars, family = binomial)
f <- function(x) predict(x, type = "link", newdata = mtcars)
p <- hypotheses(mod, hypothesis = f)
head(p)
f <- function(x) predict(x, type = "response", newdata = mtcars)
p <- hypotheses(mod, hypothesis = f)
head(p)
# Complex aggregation
# Step 1: Collapse predicted probabilities by outcome level, for each individual
# Step 2: Take the mean of the collapsed probabilities by group and `cyl`
library(dplyr)
library(MASS)
library(dplyr)
library(magrittr)
dat <- transform(mtcars, gear = factor(gear))
mod <- polr(gear ~ factor(cyl) + hp, dat)
aggregation_fun <- function(x) {
predictions(x, vcov = FALSE) %>%
mutate(group = ifelse(group %in% c("3", "4"), "3 & 4", "5")) %>%
summarize(estimate = sum(estimate), .by = c("rowid", "cyl", "group")) %>%
summarize(estimate = mean(estimate), .by = c("cyl", "group")) %>%
rename(term = cyl)
}
hypotheses(mod, hypothesis = aggregation_fun)
# Equivalence, non-inferiority, and non-superiority tests
mod <- lm(mpg ~ hp + factor(gear), data = mtcars)
p <- predictions(mod, newdata = "median")
hypotheses(p, equivalence = c(17, 18))
mfx <- avg_slopes(mod, variables = "hp")
hypotheses(mfx, equivalence = c(-.1, .1))
cmp <- avg_comparisons(mod, variables = "gear", hypothesis = ~pairwise)
hypotheses(cmp, equivalence = c(0, 10))
# joint hypotheses: character vector
model <- lm(mpg ~ as.factor(cyl) * hp, data = mtcars)
hypotheses(model, joint = c("as.factor(cyl)6:hp", "as.factor(cyl)8:hp"))
# joint hypotheses: regular expression
hypotheses(model, joint = "cyl")
# joint hypotheses: integer indices
hypotheses(model, joint = 2:3)
# joint hypotheses: different null hypotheses
hypotheses(model, joint = 2:3, hypothesis = 1)
hypotheses(model, joint = 2:3, hypothesis = 1:2)
# joint hypotheses: marginaleffects object
cmp <- avg_comparisons(model)
hypotheses(cmp, joint = "cyl")
# Multiple comparison adjustment
# p values and family-wise confidence intervals
cmp <- avg_comparisons(model)
hypotheses(cmp, multcomp = "hochberg")
```
---
## (EXPERIMENTAL) Bootstrap, Conformal, and Simulation-Based Inference {.unnumbered}
### Description
Warning: This function is experimental. It may be renamed, the user interface may change, or the functionality may migrate to arguments in other marginaleffects functions.
Apply this function to a marginaleffects object to change the inferential method used to compute uncertainty estimates.
### Usage
inferences(
x,
method,
R = 1000,
conf_type = "perc",
data_train = NULL,
data_test = NULL,
data_calib = NULL,
conformal_score = "residual_abs",
estimator = NULL,
...
)
### Arguments
x |
Object produced by one of the core marginaleffects functions, or a data frame suitable for the function supplied to the estimator argument.
|
method |
String
- "delta": delta method standard errors
- "boot" package
- "fwb": fractional weighted bootstrap
- "rsample" package
- "simulation" from a multivariate normal distribution (Krinsky & Robb, 1986)
- "conformal_split": prediction intervals using split conformal prediction (see Angelopoulos & Bates, 2022)
- "conformal_cv+": prediction intervals using cross-validation+ conformal prediction (see Barber et al., 2020)
- "conformal_full": prediction intervals using full conformal prediction (see Lei et al., 2018). Warning: This method is computationally expensive and typically much slower than split or CV+ methods.
- "conformal_quantile": prediction intervals using full conformal prediction (see Romano et al., 2020).
|
R |
Number of resamples, simulations, or cross-validation folds.
|
conf_type |
String: type of bootstrap interval to construct.
-
boot: "perc", "norm", "basic", or "bca"
-
fwb: "perc", "norm", "wald", "basic", "bc", or "bca"
-
rsample: "perc" or "bca"
-
simulation: "perc" or "wald"
|
data_train |
Data frame used to train/fit the model. If NULL, marginaleffects tries to extract the data from the original model object. Test data are inferred directly from the newdata supplied to the originating marginaleffects call (e.g., predictions()).
|
data_test |
Data frame make out of sample prediction. Only used for conformal inference. If NULL, the data frame supplied to newdata in the original marginaleffects call is used.
|
data_calib |
Data frame used for calibration in split conformal prediction.
|
conformal_score |
String. Warning: The type argument in predictions() must generate predictions which are on the same scale as the outcome variable. Typically, this means that type must be "response" or "probs".
- "residual_abs" or "residual_sq" for regression tasks (numeric outcome)
- "softmax" for classification tasks (when
predictions() returns a group columns, such as multinomial or ordinal logit models.
|
estimator |
Function that accepts a data frame, fits a model, applies a marginaleffects function, and returns the object. Only supported with method = "rsample" or method = "boot". When method = "rsample", the output must include a "term" column. This is not always the case for predictions(), in which case users may have to create the column manually.
|
... |
- If
method = "boot", additional arguments are passed to boot::boot().
- If
method = "fwb", additional arguments are passed to fwb::fwb().
- If
method = "rsample", additional arguments are passed to rsample::bootstraps(), unless the user supplies a group argument, in which case all arguments are passed to rsample::group_bootstraps().
- If
method = "conformal_full", additional arguments control the optimization process:
-
var_multiplier: multiplier for initial search bounds (default: 10)
-
max_iter: maximum iterations for root finding (default: 100)
-
tolerance: tolerance for root finding convergence (default: .Machine\$double.eps^0.25)
- If
method = "conformal_quantile", additional arguments are passed to quantregForest::quantregForest() for fitting the quantile regression forest (e.g., ntree, mtry, nodesize, nthreads).
- Additional arguments are ignored for other conformal methods (
conformal_split, conformal_cv+).
|
### Details
When method = "simulation", we conduct simulation-based inference following the method discussed in Krinsky & Robb (1986):
- Draw
R sets of simulated coefficients from a multivariate normal distribution with mean equal to the original model's estimated coefficients and variance equal to the model's variance-covariance matrix (classical, "HC3", or other).
- Use the
R sets of coefficients to compute R sets of estimands: predictions, comparisons, slopes, or hypotheses.
- Take quantiles of the resulting distribution of estimands to obtain a confidence interval (when
conf_type = "perc") and the standard deviation of simulated estimates to estimate the standard error (which is used for a Z-test and Wald confidence intervals when conf_type = "wald").
When method = "fwb", drawn weights are supplied to the model fitting function's weights argument; if the model doesn't accept non-integer weights, this method should not be used. If weights were included in the original model fit, they are extracted by weights() and multiplied by the drawn weights. These weights are supplied to the wts argument of the estimation function (e.g., comparisons()).
Warning: custom model classes are not supported by inferences() because they are not guaranteed to come with an appropriate update() method.
### Value
A marginaleffects object with simulation or bootstrap resamples and objects attached.
### References
Krinsky, I., and A. L. Robb. 1986. "On Approximating the Statistical Properties of Elasticities." Review of Economics and Statistics 68 (4): 715–9.
King, Gary, Michael Tomz, and Jason Wittenberg. "Making the most of statistical analyses: Improving interpretation and presentation." American journal of political science (2000): 347-361
Dowd, Bryan E., William H. Greene, and Edward C. Norton. "Computation of standard errors." Health services research 49.2 (2014): 731-750.
Angelopoulos, Anastasios N., and Stephen Bates. 2022. "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification." arXiv. https://doi.org/10.48550/arXiv.2107.07511.
Barber, Rina Foygel, Emmanuel J. Candes, Aaditya Ramdas, and Ryan J. Tibshirani. 2020. "Predictive Inference with the Jackknife+." arXiv. http://arxiv.org/abs/1905.02928.
Lei, Jing, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. 2018. "Distribution-Free Predictive Inference for Regression." Journal of the American Statistical Association 113 (523): 1094–1111.
Romano, Yaniv, Evan Patterson, and Emmanuel Candes. 2020. "Conformalized quantile regression." Advances in neural information processing systems 32.
### Parallel computation
The slopes() and comparisons() functions can use parallelism to
speed up computation. Operations are parallelized for the computation of
standard errors, at the model coefficient level. There is always
considerable overhead when using parallel computation, mainly involved
in passing the whole dataset to the different processes. Thus, parallel
computation is most likely to be useful when the model includes many parameters
and the dataset is relatively small.
Warning: In many cases, parallel processing will not be useful at all.
To activate parallel computation, users must load the future.apply package,
call plan() function, and set a global option.
options(marginaleffects_parallel = TRUE): parallelize delta method computation of standard errors.
options(marginaleffects_parallel_inferences = TRUE): parallelize "rsample" or "fwb" bootstrap computation in inferences().
options(marginaleffects_parallel_packages = TRUE): vector of strings with the names of modeling packages used to fit the model, ex: c("survival", "splines")
For example:
library(future.apply)
plan("multisession", workers = 4)
options(marginaleffects_parallel = FALSE)
options(marginaleffects_parallel_inferences = TRUE)
options(marginaleffects_parallel_packages = c("survival", "splines"))
slopes(model)
marginaleffects altogether, you can set a global option:
options(marginaleffects_parallel = FALSE)
newdata argument of the predictions(), comparisons(), and slopes() functions. This is useful to define where in the predictor space we want to evaluate the quantities of interest. Ex: the predicted outcome or slope for a 37 year old college graduate.
### Usage
datagrid(
...,
model = NULL,
newdata = NULL,
by = NULL,
grid_type = "mean_or_mode",
response = FALSE,
FUN = NULL,
FUN_character = NULL,
FUN_factor = NULL,
FUN_logical = NULL,
FUN_numeric = NULL,
FUN_integer = NULL,
FUN_binary = NULL,
FUN_other = NULL
)
### Arguments
... |
named arguments with vectors of values or functions for user-specified variables.
- Functions are applied to the variable in the
model dataset or newdata, and must return a vector of the appropriate type.
- Character vectors are automatically transformed to factors if necessary.
- The output will include all combinations of these variables (see Examples below.)
|
model |
Model object
|
newdata |
data.frame (one and only one of the model and newdata arguments can be used.)
|
by |
character vector with grouping variables within which FUN_* functions are applied to create "sub-grids" with unspecified variables.
|
grid_type |
character. Determines the functions to apply to each variable. The defaults can be overridden by defining individual variables explicitly in ..., or by supplying a function to one of the FUN_* arguments.
- "mean_or_mode": Character, factor, logical, and binary variables are set to their modes. Numeric, integer, and other variables are set to their means.
- "balanced": Each unique level of character, factor, logical, and binary variables are preserved. Numeric, integer, and other variables are set to their means. Warning: When there are many variables and many levels per variable, a balanced grid can be very large. In those cases, it is better to use
grid_type="mean_or_mode" and to specify the unique levels of a subset of named variables explicitly.
- "dataframe": Similar to "mean_or_mode" but creates a data frame by binding columns element-wise rather than taking the cross-product. All explicitly specified vectors must have the same length (or length 1), and the result has as many rows as the longest vector. This differs from other grid types which use
expand.grid() or data.table::CJ() to create all combinations.
- "counterfactual": the entire dataset is duplicated for each combination of the variable values specified in
.... Variables not explicitly supplied to datagrid() are set to their observed values in the original dataset.
|
response |
Logical should the response variable be included in the grid, even if it is not specified explicitly.
|
FUN |
a function to be applied to all variables in the grid. This is useful when you want to apply the same function to all variables, such as mean or median. If you specify FUN, it will override the grid_type defaults, but not other FUN_* arguments below.
|
FUN_character |
the function to be applied to character variables.
|
FUN_factor |
the function to be applied to factor variables. This only applies if the variable in the original data is a factor. For variables converted to factor in a model-fitting formula, for example, FUN_character is used.
|
FUN_logical |
the function to be applied to logical variables.
|
FUN_numeric |
the function to be applied to numeric variables.
|
FUN_integer |
the function to be applied to integer-ish variables (including columns without decimal places).
|
FUN_binary |
the function to be applied to binary variables.
|
FUN_other |
the function to be applied to other variable types.
|
### Details
If datagrid is used in a predictions(), comparisons(), or slopes() call as the
newdata argument, the model is automatically inserted in the model argument of datagrid()
call, and users do not need to specify either the model or newdata arguments. The same behavior will occur when the value supplied to newdata= is a function call which starts with "datagrid". This is intended to allow users to create convenience shortcuts like:
Warning about hierarchical grouping variables: When using the default grid_type = "mean_or_mode" with hierarchical models (such as mixed models with nested grouping factors), datagrid() may create invalid combinations of grouping variables. For example, if you have students nested within schools, or countries nested within regions, the modal values of each grouping variable may not correspond to valid nested relationships in the data. This can cause prediction errors. To avoid this issue, explicitly specify valid combinations of hierarchical grouping variables in the datagrid() call, or use grid_type = "counterfactual" to preserve the original data structure.
mod <- lm(mpg ~ am + vs + factor(cyl) + hp, mtcars)
datagrid_bal <- function(...) datagrid(..., grid_type = "balanced")
predictions(model, newdata = datagrid_bal(cyl = 4))
If users supply a model, the data used to fit that model is retrieved using
the insight::get_data function.
### Value
A data.frame in which each row corresponds to one combination of the named
predictors supplied by the user via the ... dots. Variables which are not
explicitly defined are held at their mean or mode.
### Examples
```{r, warning=FALSE, message=FALSE, eval=TRUE}
library("marginaleffects")
# The output only has 2 rows, and all the variables except `hp` are at their
# mean or mode.
datagrid(newdata = mtcars, hp = c(100, 110))
# We get the same result by feeding a model instead of a data.frame
mod <- lm(mpg ~ hp, mtcars)
datagrid(model = mod, hp = c(100, 110))
# Use in `marginaleffects` to compute "Typical Marginal Effects". When used
# in `slopes()` or `predictions()` we do not need to specify the
# `model` or `newdata` arguments.
slopes(mod, newdata = datagrid(hp = c(100, 110)))
# datagrid accepts functions
datagrid(hp = range, cyl = unique, newdata = mtcars)
comparisons(mod, newdata = datagrid(hp = fivenum))
# The full dataset is duplicated with each observation given counterfactual
# values of 100 and 110 for the `hp` variable. The original `mtcars` includes
# 32 rows, so the resulting dataset includes 64 rows.
dg <- datagrid(newdata = mtcars, hp = c(100, 110), grid_type = "counterfactual")
nrow(dg)
# We get the same result by feeding a model instead of a data.frame
mod <- lm(mpg ~ hp, mtcars)
dg <- datagrid(model = mod, hp = c(100, 110), grid_type = "counterfactual")
nrow(dg)
# Use `by` to hold variables at group-specific values
mod2 <- lm(mpg ~ hp + cyl, mtcars)
datagrid(model = mod2, hp = mean, by = "cyl")
# Use `FUN` to apply function to all variables
datagrid(model = mod2, FUN = median)
# Use `grid_type="dataframe"` for column-wise binding instead of cross-product
datagrid(model = mod2, hp = c(100, 200), cyl = c(4, 6), grid_type = "dataframe")
```
---
## Plot Conditional or Marginal Predictions {.unnumbered}
### Description
Plot predictions on the y-axis against values of one or more predictors (x-axis, colors/shapes, and facets).
The by argument is used to plot marginal predictions, that is, predictions made on the original data, but averaged by subgroups. This is analogous to using the by argument in the predictions() function.
The condition argument is used to plot conditional predictions, that is, predictions made on a user-specified grid. This is analogous to using the newdata argument and datagrid() function in a predictions() call. All variables whose values are not specified explicitly are treated as usual by datagrid(), that is, they are held at their mean or mode (or rounded mean for integers). This includes grouping variables in mixed-effects models, so analysts who fit such models may want to specify the groups of interest using the condition argument, or supply model-specific arguments to compute population-level estimates. See details below.
See the "Plots" vignette and website for tutorials and information on how to customize plots:
- https://marginaleffects.com/bonus/plot.html
- https://marginaleffects.com
### Usage
plot_predictions(
model,
condition = NULL,
by = NULL,
newdata = NULL,
type = NULL,
vcov = NULL,
conf_level = 0.95,
wts = FALSE,
transform = NULL,
points = 0,
rug = FALSE,
gray = getOption("marginaleffects_plot_gray", default = FALSE),
draw = TRUE,
...
)
### Arguments
model |
Model object
|
condition |
Conditional predictions.
- Character vector (max length 4): Names of the predictors to display.
- Named list (max length 4): List names correspond to predictors. List elements can be:
- Numeric vector
- Function which returns a numeric vector or a set of unique categorical values
- Shortcut strings for common reference values: "minmax", "quartile", "threenum"
- 1: x-axis. 2: color/shape. 3: facet (wrap if no fourth variable, otherwise cols of grid). 4: facet (rows of grid).
- Numeric variables in positions 2 and 3 are summarized by Tukey's five numbers
?stats::fivenum
|
by |
Marginal predictions
- Character vector (max length 3): Names of the categorical predictors to marginalize across.
- 1: x-axis. 2: color. 3: facets.
|
newdata |
When newdata is NULL, the grid is determined by the condition argument. When newdata is not NULL, the argument behaves in the same way as in the predictions() function. Note that the condition argument builds its own grid, so the newdata argument is ignored if the condition argument is supplied.
|
type |
string indicates the type (scale) of the predictions used to
compute contrasts or slopes. This can differ based on the model
type, but will typically be a string such as: "response", "link", "probs",
or "zero". When an unsupported string is entered, the model-specific list of
acceptable values is returned in an error message. When type is NULL, the
first entry in the error message is used by default. See the Type section in the documentation below.
|
vcov |
Type of uncertainty estimates to report (e.g., for robust standard errors). Acceptable values:
- FALSE: Do not compute standard errors. This can speed up computation considerably.
- TRUE: Unit-level standard errors using the default
vcov(model) variance-covariance matrix.
- String which indicates the kind of uncertainty estimates to return.
- Heteroskedasticity-consistent:
"HC", "HC0", "HC1", "HC2", "HC3", "HC4", "HC4m", "HC5". See ?sandwich::vcovHC
- Heteroskedasticity and autocorrelation consistent:
"HAC"
- Mixed-Models degrees of freedom: "satterthwaite", "kenward-roger"
- Other:
"NeweyWest", "KernHAC", "OPG". See the sandwich package documentation.
- "rsample", "boot", "fwb", and "simulation" are passed to the
method argument of the inferences() function. To customize the bootstrap or simulation process, call inferences() directly.
- One-sided formula which indicates the name of cluster variables (e.g.,
~unit_id). This formula is passed to the cluster argument of the sandwich::vcovCL function.
- Square covariance matrix
- Function which returns a covariance matrix (e.g.,
stats::vcov(model))
|
conf_level |
numeric value between 0 and 1. Confidence level to use to build a confidence interval.
|
wts |
logical, string or numeric: weights to use when computing average predictions, contrasts or slopes. These weights only affect the averaging in avg_*() or with the by argument, and not unit-level estimates. See ?weighted.mean
- string: column name of the weights variable in
newdata. When supplying a column name to wts, it is recommended to supply the original data (including the weights variable) explicitly to newdata.
- numeric: vector of length equal to the number of rows in the original data or in
newdata (if supplied).
- FALSE: Equal weights.
- TRUE: Extract weights from the fitted object with
insight::find_weights() and use them when taking weighted averages of estimates. Warning: newdata=datagrid() returns a single average weight, which is equivalent to using wts=FALSE
|
transform |
A function applied to unit-level adjusted predictions and confidence intervals just before the function returns results. For bayesian models, this function is applied to individual draws from the posterior distribution, before computing summaries.
|
points |
Number between 0 and 1 which controls the transparency of raw data points. 0 (default) does not display any points. Warning: The points displayed are raw data, so the resulting plot is not a "partial residual plot."
|
rug |
TRUE displays tick marks on the axes to mark the distribution of raw data.
|
gray |
FALSE grayscale or color plot
|
draw |
TRUE returns a ggplot2 plot. FALSE returns a data.frame of the underlying data.
|
... |
Additional arguments are passed to the predict() method
supplied by the modeling package.These arguments are particularly useful
for mixed-effects or bayesian models (see the online vignettes on the
marginaleffects website). Available arguments can vary from model to
model, depending on the range of supported arguments by each modeling
package. See the "Model-Specific Arguments" section of the
?slopes documentation for a non-exhaustive list of available
arguments.
|
### Value
A ggplot2 object or data frame (if draw=FALSE)
### Model-Specific Arguments
Some model types allow model-specific arguments to modify the nature of
marginal effects, predictions, marginal means, and contrasts. Please report
other package-specific predict() arguments on Github so we can add them to
the table below.
https://github.com/vincentarelbundock/marginaleffects/issues
|
Package | Class | Argument | Documentation |
brms | brmsfit | ndraws | brms::posterior_predict |
|
| | re_formula | brms::posterior_predict |
lme4 | merMod | re.form | lme4::predict.merMod |
|
| | allow.new.levels | lme4::predict.merMod |
glmmTMB | glmmTMB | re.form | glmmTMB::predict.glmmTMB |
|
| | allow.new.levels | glmmTMB::predict.glmmTMB |
|
| | zitype | glmmTMB::predict.glmmTMB |
mgcv | bam | exclude | mgcv::predict.bam |
|
| gam | exclude | mgcv::predict.gam |
robustlmm | rlmerMod | re.form | robustlmm::predict.rlmerMod |
|
| | allow.new.levels | robustlmm::predict.rlmerMod |
MCMCglmm | MCMCglmm | ndraws | |
sampleSelection | selection | part | sampleSelection::predict.selection |
|
|
### Types
The type argument determines the scale of the predictions used to compute quantities of interest with functions from the marginaleffects package. Admissible values for type depend on the model object. When users specify an incorrect value for type, marginaleffects will raise an informative error with a list of valid type values for the specific model object. The first entry in the list in that error message is the default type.
The invlink(link) is a special type defined by marginaleffects. It is available for some (but not all) models, and only for the predictions() function. With this link type, we first compute predictions on the link scale, then we use the inverse link function to backtransform the predictions to the response scale. This is useful for models with non-linear link functions as it can ensure that confidence intervals stay within desirable bounds, ex: 0 to 1 for a logit model. Note that an average of estimates with type="invlink(link)" will not always be equivalent to the average of estimates with type="response". This type is default when calling predictions(). It is available—but not default—when calling avg_predictions() or predictions() with the by argument.
Some of the most common type values are:
|
class | type |
|
Gam | invlink(link), response, link |
|
Gls | lp |
|
MCMCglmm | response |
|
bam | response, link |
|
bart | ev, ppd |
|
betareg | response, link, precision, quantile, variance |
|
bife | response, link |
|
bracl | probs |
|
brglmFit | response, link |
|
brmsfit | response, link, prediction, average |
|
brmultinom | probs, class |
|
clm | prob, cum.prob, linear.predictor |
|
clogit | expected, lp, risk, survival |
|
coxph | survival, expected, lp, risk |
|
coxph_weightit | survival, expected, lp, risk |
|
crch | response, location, scale, density |
|
fixest | invlink(link), response, link |
|
flexsurvreg | survival, response, mean, link, lp, linear, rmst, hazard, cumhaz |
|
gam | response, link |
|
geeglm | response, link |
|
glimML | response, link |
|
glm | invlink(link), response, link |
|
glm_weightit | invlink(link), probs, response, lp, link |
|
glmerMod | response, link |
|
glmgee | response |
|
glmmPQL | response, link |
|
glmmTMB | response, link, conditional, zprob, zlink, disp |
|
glmrob | response, link |
|
glmx | response |
|
hetprob | pr, xb |
|
hurdle | response, prob, count, zero |
|
hxlr | location, cumprob, scale, density |
|
iv_robust | response |
|
ivpml | pr, xb |
|
ivreg | response |
|
lda | class, posterior |
|
lm | response |
|
lm_robust | response |
|
lmerMod | response |
|
lmerModLmerTest | response |
|
lmrob | response |
|
lrm | fitted, lp, mean |
|
mblogit | response, latent, link |
|
mclogit | response, latent, link |
|
mhurdle | E, Ep, p |
|
model_fit | numeric, prob, class |
|
multinom | probs, latent |
|
multinom_weightit | probs, response, mean |
|
mvgam | response, link, expected, detection, latent_N |
|
negbin | invlink(link), response, link |
|
ols | lp |
|
oohbchoice | probability, utility |
|
ordinal_weightit | probs, response, link, lp, mean |
|
orm | fitted, mean, lp |
|
polr | probs |
|
rendo.base | response, link |
|
rlm | response |
|
selection | response, link, unconditional, conditional |
|
speedglm | response, link |
|
speedlm | response |
|
stanreg | response, link |
|
survreg | response, link, quantile |
|
svyglm | response, link |
|
svyolr | probs |
|
tobit | response, link |
|
tobit1 | expvalue, linpred, prob |
|
workflow | numeric, prob, class |
|
zeroinfl | response, prob, count, zero |
|
|
### Examples
```{r, warning=FALSE, message=FALSE, eval=FALSE}
library("marginaleffects")
mod <- lm(mpg ~ hp + wt, data = mtcars)
plot_predictions(mod, condition = "wt")
mod <- lm(mpg ~ hp * wt * am, data = mtcars)
plot_predictions(mod, condition = c("hp", "wt"))
plot_predictions(mod, condition = list("hp", wt = "threenum"))
plot_predictions(mod, condition = list("hp", wt = range))
# marginal predictions
mod <- lm(mpg ~ hp * am, data = mtcars)
plot_predictions(mod, by = "am")
# marginal predictions on a counterfactual grid
plot_predictions(mod,
by = "am",
newdata = datagrid(am = 0:1, grid_type = "counterfactual")
)
```
---
## Plot Conditional or Marginal Comparisons {.unnumbered}
### Description
Plot comparisons on the y-axis against values of one or more predictors (x-axis, colors/shapes, and facets).
The by argument is used to plot marginal comparisons, that is, comparisons made on the original data, but averaged by subgroups. This is analogous to using the by argument in the comparisons() function.
The condition argument is used to plot conditional comparisons, that is, comparisons made on a user-specified grid. This is analogous to using the newdata argument and datagrid() function in a comparisons() call. All variables whose values are not specified explicitly are treated as usual by datagrid(), that is, they are held at their mean or mode (or rounded mean for integers). This includes grouping variables in mixed-effects models, so analysts who fit such models may want to specify the groups of interest using the condition argument, or supply model-specific arguments to compute population-level estimates. See details below.
See the "Plots" vignette and website for tutorials and information on how to customize plots:
- https://marginaleffects.com/bonus/plot.html
- https://marginaleffects.com
### Usage
plot_comparisons(
model,
variables = NULL,
condition = NULL,
by = NULL,
newdata = NULL,
type = NULL,
vcov = NULL,
conf_level = 0.95,
wts = FALSE,
comparison = "difference",
transform = NULL,
rug = FALSE,
gray = getOption("marginaleffects_plot_gray", default = FALSE),
draw = TRUE,
...
)
### Arguments
model |
Model object
|
variables |
Name of the variable whose contrast we want to plot on the y-axis.
|
condition |
Conditional slopes
- Character vector (max length 4): Names of the predictors to display.
- Named list (max length 4): List names correspond to predictors. List elements can be:
- Numeric vector
- Function which returns a numeric vector or a set of unique categorical values
- Shortcut strings for common reference values: "minmax", "quartile", "threenum"
- 1: x-axis. 2: color/shape. 3: facet (wrap if no fourth variable, otherwise cols of grid). 4: facet (rows of grid).
- Numeric variables in positions 2 and 3 are summarized by Tukey's five numbers
?stats::fivenum.
|
by |
Aggregate unit-level estimates (aka, marginalize, average over). Valid inputs:
-
FALSE: return the original unit-level estimates.
-
TRUE: aggregate estimates for each term.
- Character vector of column names in
newdata or in the data frame produced by calling the function without the by argument.
- Data frame with a
by column of group labels, and merging columns shared by newdata or the data frame produced by calling the same function without the by argument.
- See examples below.
- For more complex aggregations, you can use the
FUN argument of the hypotheses() function. See that function's documentation and the Hypothesis Test vignettes on the marginaleffects website.
|
newdata |
When newdata is NULL, the grid is determined by the condition argument. When newdata is not NULL, the argument behaves in the same way as in the predictions() function. Note that the condition argument builds its own grid, so the newdata argument is ignored if the condition argument is supplied.
|
type |
string indicates the type (scale) of the predictions used to
compute contrasts or slopes. This can differ based on the model
type, but will typically be a string such as: "response", "link", "probs",
or "zero". When an unsupported string is entered, the model-specific list of
acceptable values is returned in an error message. When type is NULL, the
first entry in the error message is used by default. See the Type section in the documentation below.
|
vcov |
Type of uncertainty estimates to report (e.g., for robust standard errors). Acceptable values:
- FALSE: Do not compute standard errors. This can speed up computation considerably.
- TRUE: Unit-level standard errors using the default
vcov(model) variance-covariance matrix.
- String which indicates the kind of uncertainty estimates to return.
- Heteroskedasticity-consistent:
"HC", "HC0", "HC1", "HC2", "HC3", "HC4", "HC4m", "HC5". See ?sandwich::vcovHC
- Heteroskedasticity and autocorrelation consistent:
"HAC"
- Mixed-Models degrees of freedom: "satterthwaite", "kenward-roger"
- Other:
"NeweyWest", "KernHAC", "OPG". See the sandwich package documentation.
- "rsample", "boot", "fwb", and "simulation" are passed to the
method argument of the inferences() function. To customize the bootstrap or simulation process, call inferences() directly.
- One-sided formula which indicates the name of cluster variables (e.g.,
~unit_id). This formula is passed to the cluster argument of the sandwich::vcovCL function.
- Square covariance matrix
- Function which returns a covariance matrix (e.g.,
stats::vcov(model))
|
conf_level |
numeric value between 0 and 1. Confidence level to use to build a confidence interval.
|
wts |
logical, string or numeric: weights to use when computing average predictions, contrasts or slopes. These weights only affect the averaging in avg_*() or with the by argument, and not unit-level estimates. See ?weighted.mean
- string: column name of the weights variable in
newdata. When supplying a column name to wts, it is recommended to supply the original data (including the weights variable) explicitly to newdata.
- numeric: vector of length equal to the number of rows in the original data or in
newdata (if supplied).
- FALSE: Equal weights.
- TRUE: Extract weights from the fitted object with
insight::find_weights() and use them when taking weighted averages of estimates. Warning: newdata=datagrid() returns a single average weight, which is equivalent to using wts=FALSE
|
comparison |
How should pairs of predictions be compared? Difference, ratio, odds ratio, or user-defined functions.
- string: shortcuts to common contrast functions. Supported shortcuts strings: difference, differenceavg, differenceavgwts, dydx, eyex, eydx, dyex, dydxavg, eyexavg, eydxavg, dyexavg, dydxavgwts, eyexavgwts, eydxavgwts, dyexavgwts, ratio, ratioavg, ratioavgwts, lnratio, lnratioavg, lnratioavgwts, lnor, lnoravg, lnoravgwts, lift, liftavg, liftavgwts, expdydx, expdydxavg, expdydxavgwts
- function: accept two equal-length numeric vectors of adjusted predictions (
hi and lo) and returns a vector of contrasts of the same length, or a unique numeric value.
- See the "Comparison functions" section below for a list of common transformations and the definitions of available shortcuts.
|
transform |
string or function. Transformation applied to unit-level estimates and confidence intervals just before the function returns results. Functions must accept a vector and return a vector of the same length. Support string shortcuts: "exp", "ln"
|
rug |
TRUE displays tick marks on the axes to mark the distribution of raw data.
|
gray |
FALSE grayscale or color plot
|
draw |
TRUE returns a ggplot2 plot. FALSE returns a data.frame of the underlying data.
|
... |
Additional arguments are passed to the predict() method
supplied by the modeling package.These arguments are particularly useful
for mixed-effects or bayesian models (see the online vignettes on the
marginaleffects website). Available arguments can vary from model to
model, depending on the range of supported arguments by each modeling
package. See the "Model-Specific Arguments" section of the
?slopes documentation for a non-exhaustive list of available
arguments.
|
### Value
A ggplot2 object
### Model-Specific Arguments
Some model types allow model-specific arguments to modify the nature of
marginal effects, predictions, marginal means, and contrasts. Please report
other package-specific predict() arguments on Github so we can add them to
the table below.
https://github.com/vincentarelbundock/marginaleffects/issues
|
Package | Class | Argument | Documentation |
brms | brmsfit | ndraws | brms::posterior_predict |
|
| | re_formula | brms::posterior_predict |
lme4 | merMod | re.form | lme4::predict.merMod |
|
| | allow.new.levels | lme4::predict.merMod |
glmmTMB | glmmTMB | re.form | glmmTMB::predict.glmmTMB |
|
| | allow.new.levels | glmmTMB::predict.glmmTMB |
|
| | zitype | glmmTMB::predict.glmmTMB |
mgcv | bam | exclude | mgcv::predict.bam |
|
| gam | exclude | mgcv::predict.gam |
robustlmm | rlmerMod | re.form | robustlmm::predict.rlmerMod |
|
| | allow.new.levels | robustlmm::predict.rlmerMod |
MCMCglmm | MCMCglmm | ndraws | |
sampleSelection | selection | part | sampleSelection::predict.selection |
|
|
### Examples
```{r, warning=FALSE, message=FALSE, eval=FALSE}
library("marginaleffects")
mod <- lm(mpg ~ hp * drat * factor(am), data = mtcars)
plot_comparisons(mod, variables = "hp", condition = "drat")
plot_comparisons(mod, variables = "hp", condition = c("drat", "am"))
plot_comparisons(mod, variables = "hp", condition = list("am", "drat" = 3:5))
plot_comparisons(mod, variables = "am", condition = list("hp", "drat" = range))
plot_comparisons(mod, variables = "am", condition = list("hp", "drat" = "threenum"))
# marginal comparisons
plot_comparisons(mod, variables = "hp", by = "am")
# marginal comparisons on a counterfactual grid
plot_comparisons(mod,
variables = "hp",
by = "am",
newdata = datagrid(am = 0:1, grid_type = "counterfactual")
)
```
---
## Plot Conditional or Marginal Slopes {.unnumbered}
### Description
Plot slopes on the y-axis against values of one or more predictors (x-axis, colors/shapes, and facets).
The by argument is used to plot marginal slopes, that is, slopes made on the original data, but averaged by subgroups. This is analogous to using the by argument in the slopes() function.
The condition argument is used to plot conditional slopes, that is, slopes computed on a user-specified grid. This is analogous to using the newdata argument and datagrid() function in a slopes() call. All variables whose values are not specified explicitly are treated as usual by datagrid(), that is, they are held at their mean or mode (or rounded mean for integers). This includes grouping variables in mixed-effects models, so analysts who fit such models may want to specify the groups of interest using the condition argument, or supply model-specific arguments to compute population-level estimates. See details below.
See the "Plots" vignette and website for tutorials and information on how to customize plots:
- https://marginaleffects.com/bonus/plot.html
- https://marginaleffects.com
### Usage
plot_slopes(
model,
variables = NULL,
condition = NULL,
by = NULL,
newdata = NULL,
type = NULL,
vcov = NULL,
conf_level = 0.95,
wts = FALSE,
slope = "dydx",
rug = FALSE,
gray = getOption("marginaleffects_plot_gray", default = FALSE),
draw = TRUE,
...
)
### Arguments
model |
Model object
|
variables |
Name of the variable whose marginal effect (slope) we want to plot on the y-axis.
|
condition |
Conditional slopes
- Character vector (max length 4): Names of the predictors to display.
- Named list (max length 4): List names correspond to predictors. List elements can be:
- Numeric vector
- Function which returns a numeric vector or a set of unique categorical values
- Shortcut strings for common reference values: "minmax", "quartile", "threenum"
- 1: x-axis. 2: color/shape. 3: facet (wrap if no fourth variable, otherwise cols of grid). 4: facet (rows of grid).
- Numeric variables in positions 2 and 3 are summarized by Tukey's five numbers
?stats::fivenum.
|
by |
Aggregate unit-level estimates (aka, marginalize, average over). Valid inputs:
-
FALSE: return the original unit-level estimates.
-
TRUE: aggregate estimates for each term.
- Character vector of column names in
newdata or in the data frame produced by calling the function without the by argument.
- Data frame with a
by column of group labels, and merging columns shared by newdata or the data frame produced by calling the same function without the by argument.
- See examples below.
- For more complex aggregations, you can use the
FUN argument of the hypotheses() function. See that function's documentation and the Hypothesis Test vignettes on the marginaleffects website.
|
newdata |
When newdata is NULL, the grid is determined by the condition argument. When newdata is not NULL, the argument behaves in the same way as in the predictions() function. Note that the condition argument builds its own grid, so the newdata argument is ignored if the condition argument is supplied.
|
type |
string indicates the type (scale) of the predictions used to
compute contrasts or slopes. This can differ based on the model
type, but will typically be a string such as: "response", "link", "probs",
or "zero". When an unsupported string is entered, the model-specific list of
acceptable values is returned in an error message. When type is NULL, the
first entry in the error message is used by default. See the Type section in the documentation below.
|
vcov |
Type of uncertainty estimates to report (e.g., for robust standard errors). Acceptable values:
- FALSE: Do not compute standard errors. This can speed up computation considerably.
- TRUE: Unit-level standard errors using the default
vcov(model) variance-covariance matrix.
- String which indicates the kind of uncertainty estimates to return.
- Heteroskedasticity-consistent:
"HC", "HC0", "HC1", "HC2", "HC3", "HC4", "HC4m", "HC5". See ?sandwich::vcovHC
- Heteroskedasticity and autocorrelation consistent:
"HAC"
- Mixed-Models degrees of freedom: "satterthwaite", "kenward-roger"
- Other:
"NeweyWest", "KernHAC", "OPG". See the sandwich package documentation.
- "rsample", "boot", "fwb", and "simulation" are passed to the
method argument of the inferences() function. To customize the bootstrap or simulation process, call inferences() directly.
- One-sided formula which indicates the name of cluster variables (e.g.,
~unit_id). This formula is passed to the cluster argument of the sandwich::vcovCL function.
- Square covariance matrix
- Function which returns a covariance matrix (e.g.,
stats::vcov(model))
|
conf_level |
numeric value between 0 and 1. Confidence level to use to build a confidence interval.
|
wts |
logical, string or numeric: weights to use when computing average predictions, contrasts or slopes. These weights only affect the averaging in avg_*() or with the by argument, and not unit-level estimates. See ?weighted.mean
- string: column name of the weights variable in
newdata. When supplying a column name to wts, it is recommended to supply the original data (including the weights variable) explicitly to newdata.
- numeric: vector of length equal to the number of rows in the original data or in
newdata (if supplied).
- FALSE: Equal weights.
- TRUE: Extract weights from the fitted object with
insight::find_weights() and use them when taking weighted averages of estimates. Warning: newdata=datagrid() returns a single average weight, which is equivalent to using wts=FALSE
|
slope |
string indicates the type of slope or (semi-)elasticity to compute:
- "dydx": dY/dX
- "eyex": dY/dX * Y / X
- "eydx": dY/dX * Y
- "dyex": dY/dX / X
- Y is the predicted value of the outcome; X is the observed value of the predictor.
|
rug |
TRUE displays tick marks on the axes to mark the distribution of raw data.
|
gray |
FALSE grayscale or color plot
|
draw |
TRUE returns a ggplot2 plot. FALSE returns a data.frame of the underlying data.
|
... |
Additional arguments are passed to the predict() method
supplied by the modeling package.These arguments are particularly useful
for mixed-effects or bayesian models (see the online vignettes on the
marginaleffects website). Available arguments can vary from model to
model, depending on the range of supported arguments by each modeling
package. See the "Model-Specific Arguments" section of the
?slopes documentation for a non-exhaustive list of available
arguments.
|
### Value
A ggplot2 object
### Model-Specific Arguments
Some model types allow model-specific arguments to modify the nature of
marginal effects, predictions, marginal means, and contrasts. Please report
other package-specific predict() arguments on Github so we can add them to
the table below.
https://github.com/vincentarelbundock/marginaleffects/issues
|
Package | Class | Argument | Documentation |
brms | brmsfit | ndraws | brms::posterior_predict |
|
| | re_formula | brms::posterior_predict |
lme4 | merMod | re.form | lme4::predict.merMod |
|
| | allow.new.levels | lme4::predict.merMod |
glmmTMB | glmmTMB | re.form | glmmTMB::predict.glmmTMB |
|
| | allow.new.levels | glmmTMB::predict.glmmTMB |
|
| | zitype | glmmTMB::predict.glmmTMB |
mgcv | bam | exclude | mgcv::predict.bam |
|
| gam | exclude | mgcv::predict.gam |
robustlmm | rlmerMod | re.form | robustlmm::predict.rlmerMod |
|
| | allow.new.levels | robustlmm::predict.rlmerMod |
MCMCglmm | MCMCglmm | ndraws | |
sampleSelection | selection | part | sampleSelection::predict.selection |
|
|
### Examples
```{r, warning=FALSE, message=FALSE, eval=FALSE}
library("marginaleffects")
mod <- lm(mpg ~ hp * drat * factor(am), data = mtcars)
plot_slopes(mod, variables = "hp", condition = "drat")
plot_slopes(mod, variables = "hp", condition = c("drat", "am"))
plot_slopes(mod, variables = "hp", condition = list("am", "drat" = 3:5))
plot_slopes(mod, variables = "am", condition = list("hp", "drat" = range))
plot_slopes(mod, variables = "am", condition = list("hp", "drat" = "threenum"))
# marginal slopes
plot_slopes(mod, variables = "hp", by = "am")
# marginal slopes on a counterfactual grid
plot_slopes(mod,
variables = "hp",
by = "am",
newdata = datagrid(am = 0:1, grid_type = "counterfactual")
)
```
---
## EXPERIMENTAL – Enable Automatic Differentiation with JAX {.unnumbered}
### Description
This function enables or disables automatic differentiation using the JAX
package in Python, which can considerably speed up and increase the accuracy
of standard errors when a model includes many parameters.
### Usage
autodiff(autodiff = NULL, install = FALSE)
### Arguments
autodiff |
Logical flag. If TRUE, enables automatic differentiation
with JAX. If FALSE (default), disables automatic differentiation and
reverts to finite difference methods.
|
install |
Logical flag. If TRUE, installs the marginaleffects
Python package via reticulate::py_install(). Default is FALSE. This is
only necessary if you are self-managing a Python installation.
|
### Details
Automatic differentiation needs to be enabled once per session.
When autodiff = TRUE, this function:
- Imports the
marginaleffects.autodiff Python module via reticulate::import()
- Sets the internal jacobian function to use JAX-based automatic differentiation
- Provides faster and more accurate gradient computation for supported models
- Falls back on the default finite difference method for unsupported models and calls.
Currently supports:
- Model types:
lm, glm, ols
- Functions:
predictions() and comparisons(), along with avg_ and plot_ variants.
-
type: "response" or "link"
-
by: TRUE, FALSE, or character vector.
-
comparison: "difference" and "ratio"
For unsupported models or options, the function automatically falls back to
the default finite difference method.
### Value
No return value. Called for side effects of enabling/disabling
automatic differentiation.
### Python Configuration
By default, no manual configuration of Python should be necessary. On most
machines, unless you have explicitly configured reticulate, reticulate
defaults to an automatically managed ephemeral virtual environment with all
Python requirements declared via reticulate::py_require().
If you prefer to use a manually managed Python installation, you can direct
reticulate and specify which Python executable or environment to use.
reticulate selects a Python installation using its Order of Discovery.
As a convenience autodiff(install=TRUE) will install the marginaleffects Python
package in a self-managed virtual environment.
To specify an alternate Python version:
library(reticulate)
use_python("/usr/local/bin/python")
autodiff().
### Examples
```{r, warning=FALSE, message=FALSE, eval=FALSE}
library("marginaleffects")
# Install the Python package (only needed once)
autodiff(install = TRUE)
# Enable automatic differentiation
autodiff(TRUE)
# Fit a model and compute marginal effects
mod <- glm(am ~ hp + wt, data = mtcars, family = binomial)
avg_comparisons(mod) # Will use JAX for faster computation
# Disable automatic differentiation
autodiff(FALSE)
```
---
## alias to get_draws() keep forever for backward compatibility with JSS {.unnumbered}
### Description
alias to get_draws() keep forever for backward compatibility with JSS
### Usage
posterior_draws(x, shape = "long")
---
## Download and Read Datasets from marginaleffects or Rdatasets {.unnumbered}
### Description
Downloads a dataset from the marginaleffects or the Rdatasets archives, and return it as a data frame. Opens the documentation as an HTML page. Search available datasets.
https://vincentarelbundock.github.io/Rdatasets/
### Usage
get_dataset(dataset = "thornton", package = NULL, docs = FALSE, search = NULL)
### Arguments
dataset |
String. Name of the dataset to download.
-
marginaleffects archive: affairs, airbnb, ces_demographics, ces_survey, immigration, lottery, military, thornton, factorial_01, interaction_01, interaction_02, interaction_03, interaction_04, polynomial_01, polynomial_02
- Rdatasets archive: The name of a dataset listed on the Rdatasets index. See the website or the
search argument.
|
package |
String. Package name that originally published the data.
|
docs |
Logical. If TRUE open the documentation using getOption("viewer") or the Rstudio viewer.
|
search |
Regular expression. Download the dataset index from Rdatasets; search the "Package", "Item", and "Title" columns; and return the matching rows.
|
### Value
A data frame containing the dataset.
### Examples
```{r, warning=FALSE, message=FALSE, eval=FALSE}
library("marginaleffects")
dat <- get_dataset("Titanic", "Stat2Data")
head(dat)
get_dataset(search = "(?i)titanic")
# View documentation in the browser
get_dataset("Titanic", "Stat2Data", docs = TRUE)
```
---
## Print marginaleffects objects {.unnumbered}
### Description
This function controls the text which is printed to the console when one of the core marginalefffects functions is called and the object is returned: predictions(), comparisons(), slopes(), hypotheses(), avg_predictions(), avg_comparisons(), avg_slopes().
All of those functions return standard data frames. Columns can be extracted by name, predictions(model)\$estimate, and all the usual data manipulation functions work out-of-the-box: colnames(), head(), subset(), dplyr::filter(), dplyr::arrange(), etc.
Some of the data columns are not printed by default. You can disable pretty printing and print the full results as a standard data frame using the style argument or by applying as.data.frame() on the object. See examples below.
### Usage
## S3 method for class 'marginaleffects'
print(
x,
style = getOption("marginaleffects_print_style", default = "summary"),
digits = getOption("marginaleffects_print_digits", default = 3),
p_eps = getOption("marginaleffects_print_p_eps", default = 0.001),
topn = getOption("marginaleffects_print_topn", default = 5),
nrows = getOption("marginaleffects_print_nrows", default = 30),
ncols = getOption("marginaleffects_print_ncols", default = 30),
type = getOption("marginaleffects_print_type", default = TRUE),
column_names = getOption("marginaleffects_print_column_names", default = FALSE),
...
)
### Arguments
x |
An object produced by one of the marginaleffects package functions.
|
style |
"summary", "data.frame", or "tinytable"
|
digits |
The number of digits to display.
|
p_eps |
p values smaller than this number are printed in "<0.001" style.
|
topn |
The number of rows to be printed from the beginning and end of tables with more than nrows rows.
|
nrows |
The number of rows which will be printed before truncation.
|
ncols |
The maximum number of column names to display at the bottom of the printed output.
|
type |
boolean: should the type be printed?
|
column_names |
boolean: should the column names be printed?
|
... |
Other arguments are currently ignored.
|
### Examples
```{r, warning=FALSE, message=FALSE, eval=TRUE}
library("marginaleffects")
library(marginaleffects)
mod <- lm(mpg ~ hp + am + factor(gear), data = mtcars)
p <- predictions(mod, by = c("am", "gear"))
p
subset(p, am == 1)
print(p, style = "data.frame")
data.frame(p)
```
---
# Function Reference (Python)
---
# `predictions()` {.unnumbered}
`predictions()` and `avg_predictions()` predict outcomes using a fitted model on a specified scale for given combinations of values of predictor variables, such as their observed values, means, or factor levels (reference grid).
* `predictions()`: unit-level (conditional) estimates.
* `avg_predictions()`: average (marginal) estimates.
See the package website and vignette for examples:
- https://marginaleffects.com/chapters/predictions.html
- https://marginaleffects.com
## Parameters {.unnumbered}
`model`: (model object) Object fitted using the `statsmodels` formula API.
`variables`: (str, list, dictionary) Specifies what variables (columns) to vary in order to make the prediction.
- `None`: predictions are computed for all regressors in the model object (can be slow). Acceptable values depend on the variable type. See the examples below.
- List[str] or str: List of variable names to compute predictions for.
- Dictionary: keys identify the subset of variables of interest, and values define the type of contrast to compute. Acceptable values depend on the variable type:
- Categorical variables:
* "reference": Each factor level is compared to the factor reference (base) level
* "all": All combinations of observed levels
* "sequential": Each factor level is compared to the previous factor level
* "pairwise": Each factor level is compared to all other levels
* "minmax": The highest and lowest levels of a factor.
* "revpairwise", "revreference", "revsequential": inverse of the corresponding hypotheses.
* Vector of length 2 with the two values to compare.
- Boolean variables:
* `None`: contrast between True and False
- Numeric variables:
* Numeric of length 1: Contrast for a gap of `x`, computed at the observed value plus and minus `x / 2`. For example, estimating a `+1` contrast compares adjusted predictions when the regressor is equal to its observed value minus 0.5 and its observed value plus 0.5.
* Numeric of length equal to the number of rows in `newdata`: Same as above, but the contrast can be customized for each row of `newdata`.
* Numeric vector of length 2: Contrast between the 2nd element and the 1st element of the `x` vector.
* Data frame with the same number of rows as `newdata`, with two columns of "low" and "high" values to compare.
* Function which accepts a numeric vector and returns a data frame with two columns of "low" and "high" values to compare. See examples below.
* "iqr": Contrast across the interquartile range of the regressor.
* "sd": Contrast across one standard deviation around the regressor mean.
* "2sd": Contrast across two standard deviations around the regressor mean.
* "minmax": Contrast between the maximum and the minimum values of the regressor.
- Examples:
+ `variables = "gear" : "pairwise", "hp" : 10`
+ `variables = "gear" : "sequential", "hp" : [100, 120]`
`newdata`: (None, DataFrame, str) Data frame or string specifying where statistics are evaluated in the predictor space.
- None: Compute predictions at each observed value in the original dataset (empirical distribution)
- Dataframe: should be created with datagrid() function
- str:
* "mean": Compute predictions at the mean of the regressor
* "median": Compute predictions at the median of the regressor
* "balanced": Compute predictions on a balanced grid with every combination of categories and numeric variables held at their means.
* "tukey": Probably NotImplemented
* "grid": Probably NotImplemented
`by`: (bool, List[str], optional) A logical value or a list of column names in `newdata`.
- `True`: estimate is aggregated across the whole dataset.
- list: estimates are aggregated for each unique combination of values in the columns.
`transform`: (function) Function specifying a transformation applied to unit-level estimates and confidence intervals just before the function returns results. Functions must accept a full column (series) of a Polars data frame and return a corresponding series of the same length. Ex:
- `transform = numpy.exp`
- `transform = lambda x: x.exp()`
- `transform = lambda x: x.map_elements()`
`hypothesis`: (str, int, float, list of str, numpy array) Specifies a hypothesis test or custom contrast
* Number to specify the null hypothesis.
* Numpy array with a number of rows equal to the number of estimates.
* String equation with an equal sign and estimate number in b0, b1, b2, etc. format.
- "b0 = b1"
- "b0 - (b1 + b2) = 0"
* Two-side formula like "ratio ~ reference"
- Left-hand side: "ratio", "difference"
- Right-hand side: 'reference', 'sequential', 'pairwise', 'revreference', 'revsequential', 'revpairwise'
- int, float: The null hypothesis used in the computation of Z and p-values (before applying transform)
- str:
* equation specifying linear or non-linear hypothesis tests. Use the names of the model variables, or use `b0`, `b1` to identify the position of each parameter. The `b*` wildcard can be used to test hypotheses on all estimates. Examples:
- `hp = drat`
- `hp + drat = 12`
- `b0 + b1 + b2 = 0`
- `b* / b0 = 1`
* one of the following hypothesis test strings:
- `pairwise` and `revpairwise`: pairwise differences between estimates in each row.
- `reference` and `revreference`: differences between the estimates in each row and the estimate in the first row.
- `sequential` and `revsequential`: differences between an estimate and the estimate in the next row.
- list of strings: Multiple hypotheses evaluated in sequence, each processed as if passed individually. The resulting rows are stacked in the order supplied. Example: `["b1 - b0 = 0", "b2 = 1"]`.
- numpy.ndarray: Each column is a vector of weights. The output is the dot product between these vectors of weights and the vectors of estimates. e.g. `hypothesis=np.array([[1, 1, 2], [2, 2, 3]]).T`
- See the Examples section and the vignette: https://marginaleffects.com/chapters/hypothesis.html
`wts`: (str, optional) Column name of weights to use for marginalization. Must be a column in `newdata`.
`vcov`: (bool, np.ndarray, default=True) Type of uncertainty estimates to report (e.g. for robust standard errors). Acceptable values are:
- `True`: Use the model's default covariance matrix.
- `False`: Do not compute standard errors.
- String: Literal indicating the kind of uncertainty estimates to return:
- Heteroskedasticity-consistent: `"HC0"`, `"HC1"`, `"HC2"`, `"HC3"`.
- np.ndarray: A custom square covariance matrix.
`equivalence`: (list, optional) List of 2 numeric float values specifying the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. See the Details section below.
`conf_level`: (float, default=0.95) Numeric value specifying the confidence level for the confidence intervals.
`eps_vcov`: (float) optional custom value for the finite difference approximation of the jacobian matrix. By default, the function uses the square root of the machine epsilon.
## Returns {.unnumbered}
A Polars DataFrame with (some of) the following columns:
- `term`: the name of the variable.
- `contrast`: the comparison method used.
- `estimate`: the estimated contrast, difference, ratio, or other transformation between pairs of predictions.
- `std_error`: the standard error of the estimate.
- `statistic`: the test statistic (estimate / std.error).
- `p_value`: the p-value of the test.
- `s_value`: Shannon transform of the p value.
- `conf_low`: the lower confidence interval bound.
- `conf_high`: the upper confidence interval bound.
- `pred_low`: the lower prediction interval bound.
- `pred_high`: the upper prediction interval bound.
## Examples {.unnumbered}
```py
from marginaleffects import *
import statsmodels.api as sm
import statsmodels.formula.api as smf
data = get_dataset("thornton")
mod = smf.ols("outcome ~ incentive + distance", data).fit()
predictions(mod)
avg_predictions(mod)
predictions(mod, by = "village")
avg_predictions(mod, by = "village")
predictions(mod, hypothesis = 3)
avg_predictions(mod, hypothesis = 3)
```
## Details {.unnumbered}
### Two-One-Sided Test (TOST) of Equivalence {.unnumbered}
The `equivalence` argument specifies the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. The first element specifies the lower bound, and the second element specifies the upper bound. If `None`, equivalence tests are not performed.
### Order of operations. {.unnumbered}
Behind the scenes, the arguments of `marginaleffects` functions are evaluated in this order:
1. `newdata`
2. `variables`
3. `comparison` and `slope`
4. `by`
5. `vcov`
6. `hypothesis`
7. `transform`
---
# `predictions()` {.unnumbered}
`predictions()` and `avg_predictions()` predict outcomes using a fitted model on a specified scale for given combinations of values of predictor variables, such as their observed values, means, or factor levels (reference grid).
* `predictions()`: unit-level (conditional) estimates.
* `avg_predictions()`: average (marginal) estimates.
See the package website and vignette for examples:
- https://marginaleffects.com/chapters/predictions.html
- https://marginaleffects.com
## Parameters {.unnumbered}
`model`: (model object) Object fitted using the `statsmodels` formula API.
`variables`: (str, list, dictionary) Specifies what variables (columns) to vary in order to make the prediction.
- `None`: predictions are computed for all regressors in the model object (can be slow). Acceptable values depend on the variable type. See the examples below.
- List[str] or str: List of variable names to compute predictions for.
- Dictionary: keys identify the subset of variables of interest, and values define the type of contrast to compute. Acceptable values depend on the variable type:
- Categorical variables:
* "reference": Each factor level is compared to the factor reference (base) level
* "all": All combinations of observed levels
* "sequential": Each factor level is compared to the previous factor level
* "pairwise": Each factor level is compared to all other levels
* "minmax": The highest and lowest levels of a factor.
* "revpairwise", "revreference", "revsequential": inverse of the corresponding hypotheses.
* Vector of length 2 with the two values to compare.
- Boolean variables:
* `None`: contrast between True and False
- Numeric variables:
* Numeric of length 1: Contrast for a gap of `x`, computed at the observed value plus and minus `x / 2`. For example, estimating a `+1` contrast compares adjusted predictions when the regressor is equal to its observed value minus 0.5 and its observed value plus 0.5.
* Numeric of length equal to the number of rows in `newdata`: Same as above, but the contrast can be customized for each row of `newdata`.
* Numeric vector of length 2: Contrast between the 2nd element and the 1st element of the `x` vector.
* Data frame with the same number of rows as `newdata`, with two columns of "low" and "high" values to compare.
* Function which accepts a numeric vector and returns a data frame with two columns of "low" and "high" values to compare. See examples below.
* "iqr": Contrast across the interquartile range of the regressor.
* "sd": Contrast across one standard deviation around the regressor mean.
* "2sd": Contrast across two standard deviations around the regressor mean.
* "minmax": Contrast between the maximum and the minimum values of the regressor.
- Examples:
+ `variables = "gear" : "pairwise", "hp" : 10`
+ `variables = "gear" : "sequential", "hp" : [100, 120]`
`newdata`: (None, DataFrame, str) Data frame or string specifying where statistics are evaluated in the predictor space.
- None: Compute predictions at each observed value in the original dataset (empirical distribution)
- Dataframe: should be created with datagrid() function
- str:
* "mean": Compute predictions at the mean of the regressor
* "median": Compute predictions at the median of the regressor
* "balanced": Compute predictions on a balanced grid with every combination of categories and numeric variables held at their means.
* "tukey": Probably NotImplemented
* "grid": Probably NotImplemented
`by`: (bool, List[str], optional) A logical value or a list of column names in `newdata`.
- `True`: estimate is aggregated across the whole dataset.
- list: estimates are aggregated for each unique combination of values in the columns.
`transform`: (function) Function specifying a transformation applied to unit-level estimates and confidence intervals just before the function returns results. Functions must accept a full column (series) of a Polars data frame and return a corresponding series of the same length. Ex:
- `transform = numpy.exp`
- `transform = lambda x: x.exp()`
- `transform = lambda x: x.map_elements()`
`hypothesis`: (str, int, float, list of str, numpy array) Specifies a hypothesis test or custom contrast
* Number to specify the null hypothesis.
* Numpy array with a number of rows equal to the number of estimates.
* String equation with an equal sign and estimate number in b0, b1, b2, etc. format.
- "b0 = b1"
- "b0 - (b1 + b2) = 0"
* Two-side formula like "ratio ~ reference"
- Left-hand side: "ratio", "difference"
- Right-hand side: 'reference', 'sequential', 'pairwise', 'revreference', 'revsequential', 'revpairwise'
- int, float: The null hypothesis used in the computation of Z and p-values (before applying transform)
- str:
* equation specifying linear or non-linear hypothesis tests. Use the names of the model variables, or use `b0`, `b1` to identify the position of each parameter. The `b*` wildcard can be used to test hypotheses on all estimates. Examples:
- `hp = drat`
- `hp + drat = 12`
- `b0 + b1 + b2 = 0`
- `b* / b0 = 1`
* one of the following hypothesis test strings:
- `pairwise` and `revpairwise`: pairwise differences between estimates in each row.
- `reference` and `revreference`: differences between the estimates in each row and the estimate in the first row.
- `sequential` and `revsequential`: differences between an estimate and the estimate in the next row.
- list of strings: Multiple hypotheses evaluated in sequence, each processed as if passed individually. The resulting rows are stacked in the order supplied. Example: `["b1 - b0 = 0", "b2 = 1"]`.
- numpy.ndarray: Each column is a vector of weights. The output is the dot product between these vectors of weights and the vectors of estimates. e.g. `hypothesis=np.array([[1, 1, 2], [2, 2, 3]]).T`
- See the Examples section and the vignette: https://marginaleffects.com/chapters/hypothesis.html
`wts`: (str, optional) Column name of weights to use for marginalization. Must be a column in `newdata`.
`vcov`: (bool, np.ndarray, default=True) Type of uncertainty estimates to report (e.g. for robust standard errors). Acceptable values are:
- `True`: Use the model's default covariance matrix.
- `False`: Do not compute standard errors.
- String: Literal indicating the kind of uncertainty estimates to return:
- Heteroskedasticity-consistent: `"HC0"`, `"HC1"`, `"HC2"`, `"HC3"`.
- np.ndarray: A custom square covariance matrix.
`equivalence`: (list, optional) List of 2 numeric float values specifying the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. See the Details section below.
`conf_level`: (float, default=0.95) Numeric value specifying the confidence level for the confidence intervals.
`eps_vcov`: (float) optional custom value for the finite difference approximation of the jacobian matrix. By default, the function uses the square root of the machine epsilon.
## Returns {.unnumbered}
A Polars DataFrame with (some of) the following columns:
- `term`: the name of the variable.
- `contrast`: the comparison method used.
- `estimate`: the estimated contrast, difference, ratio, or other transformation between pairs of predictions.
- `std_error`: the standard error of the estimate.
- `statistic`: the test statistic (estimate / std.error).
- `p_value`: the p-value of the test.
- `s_value`: Shannon transform of the p value.
- `conf_low`: the lower confidence interval bound.
- `conf_high`: the upper confidence interval bound.
- `pred_low`: the lower prediction interval bound.
- `pred_high`: the upper prediction interval bound.
## Examples {.unnumbered}
```py
from marginaleffects import *
import statsmodels.api as sm
import statsmodels.formula.api as smf
data = get_dataset("thornton")
mod = smf.ols("outcome ~ incentive + distance", data).fit()
predictions(mod)
avg_predictions(mod)
predictions(mod, by = "village")
avg_predictions(mod, by = "village")
predictions(mod, hypothesis = 3)
avg_predictions(mod, hypothesis = 3)
```
## Details {.unnumbered}
### Two-One-Sided Test (TOST) of Equivalence {.unnumbered}
The `equivalence` argument specifies the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. The first element specifies the lower bound, and the second element specifies the upper bound. If `None`, equivalence tests are not performed.
### Order of operations. {.unnumbered}
Behind the scenes, the arguments of `marginaleffects` functions are evaluated in this order:
1. `newdata`
2. `variables`
3. `comparison` and `slope`
4. `by`
5. `vcov`
6. `hypothesis`
7. `transform`
---
# `comparisons()` {.unnumbered}
`comparisons()` and `avg_comparisons()` are functions for predicting the outcome variable at different regressor values and comparing those predictions by computing a difference, ratio, or some other function. These functions can return many quantities of interest, such as contrasts, differences, risk ratios, changes in log odds, lift, slopes, elasticities, average treatment effect (on the treated or untreated), etc.
* `comparisons()`: unit-level (conditional) estimates.
* `avg_comparisons()`: average (marginal) estimates.
See the package website and vignette for examples:
* https://marginaleffects.com/chapters/comparisons.html
* https://marginaleffects.com
## Parameters {.unnumbered}
`model`: (model object) Object fitted using the `statsmodels` formula API.
`variables`: (str, list, dictionary) Specifies what variables (columns) to vary in order to make the comparison.
- `None`: comparisons are computed for all regressors in the model object (can be slow). Acceptable values depend on the variable type. See the examples below.
- List[str] or str: List of variable names to compute comparisons for.
- Dictionary: keys identify the subset of variables of interest, and values define the type of contrast to compute. Acceptable values depend on the variable type:
- Categorical variables:
* "reference": Each factor level is compared to the factor reference (base) level
* "all": All combinations of observed levels
* "sequential": Each factor level is compared to the previous factor level
* "pairwise": Each factor level is compared to all other levels
* "minmax": The highest and lowest levels of a factor.
* "revpairwise", "revreference", "revsequential": inverse of the corresponding hypotheses.
* Vector of length 2 with the two values to compare.
- Boolean variables:
* `None`: contrast between True and False
- Numeric variables:
* Numeric of length 1: Contrast for a gap of `x`, computed at the observed value plus and minus `x / 2`. For example, estimating a `+1` contrast compares adjusted predictions when the regressor is equal to its observed value minus 0.5 and its observed value plus 0.5.
* Numeric of length equal to the number of rows in `newdata`: Same as above, but the contrast can be customized for each row of `newdata`.
* Numeric vector of length 2: Contrast between the 2nd element and the 1st element of the `x` vector.
* Data frame with the same number of rows as `newdata`, with two columns of "low" and "high" values to compare.
* Function which accepts a numeric vector and returns a data frame with two columns of "low" and "high" values to compare. See examples below.
* "iqr": Contrast across the interquartile range of the regressor.
* "sd": Contrast across one standard deviation around the regressor mean.
* "2sd": Contrast across two standard deviations around the regressor mean.
* "minmax": Contrast between the maximum and the minimum values of the regressor.
- Examples:
+ `variables = "gear" : "pairwise", "hp" : 10`
+ `variables = "gear" : "sequential", "hp" : [100, 120]`
`newdata`: (None, DataFrame, str) Data frame or string specifying where statistics are evaluated in the predictor space.
- None: Compute comparisons at each observed value in the original dataset (empirical distribution)
- Dataframe: should be created with datagrid() function
- str:
* "mean": Compute comparisons at the mean of the regressor
* "median": Compute comparisons at the median of the regressor
* "balanced": Compute comparisons on a balanced grid with every combination of categories and numeric variables held at their means.
* "tukey": Probably NotImplemented
* "grid": Probably NotImplemented
* `comparison`: (str or callable) String specifying how pairs of predictions should be compared, or a callable function to compute custom estimates. See the Comparisons section below for definitions of each transformation.
* Acceptable strings: difference, differenceavg, differenceavgwts, dydx, eyex, eydx, dyex, dydxavg, eyexavg, eydxavg, dyexavg, dydxavgwts, eyexavgwts, eydxavgwts, dyexavgwts, ratio, ratioavg, ratioavgwts, lnratio, lnratioavg, lnratioavgwts, lnor, lnoravg, lnoravgwts, lift, liftavg, liftavg, expdydx, expdydxavg, expdydxavgwts
* Callable: A function that takes `hi`, `lo`, `eps`, `x`, `y`, and `w` as arguments and returns a numeric array. This allows computing custom comparisons like `lambda hi, lo, eps, x, y, w: hi / lo` for ratios or `lambda hi, lo, eps, x, y, w: (hi - lo) / lo * 100` for percent changes.
`by`: (bool, List[str], optional) A logical value or a list of column names in `newdata`.
- `True`: estimate is aggregated across the whole dataset.
- list: estimates are aggregated for each unique combination of values in the columns.
`transform`: (function) Function specifying a transformation applied to unit-level estimates and confidence intervals just before the function returns results. Functions must accept a full column (series) of a Polars data frame and return a corresponding series of the same length. Ex:
- `transform = numpy.exp`
- `transform = lambda x: x.exp()`
- `transform = lambda x: x.map_elements()`
`hypothesis`: (str, int, float, list of str, numpy array) Specifies a hypothesis test or custom contrast
* Number to specify the null hypothesis.
* Numpy array with a number of rows equal to the number of estimates.
* String equation with an equal sign and estimate number in b0, b1, b2, etc. format.
- "b0 = b1"
- "b0 - (b1 + b2) = 0"
* Two-side formula like "ratio ~ reference"
- Left-hand side: "ratio", "difference"
- Right-hand side: 'reference', 'sequential', 'pairwise', 'revreference', 'revsequential', 'revpairwise'
- int, float: The null hypothesis used in the computation of Z and p-values (before applying transform)
- str:
* equation specifying linear or non-linear hypothesis tests. Use the names of the model variables, or use `b0`, `b1` to identify the position of each parameter. The `b*` wildcard can be used to test hypotheses on all estimates. Examples:
- `hp = drat`
- `hp + drat = 12`
- `b0 + b1 + b2 = 0`
- `b* / b0 = 1`
* one of the following hypothesis test strings:
- `pairwise` and `revpairwise`: pairwise differences between estimates in each row.
- `reference` and `revreference`: differences between the estimates in each row and the estimate in the first row.
- `sequential` and `revsequential`: differences between an estimate and the estimate in the next row.
- list of strings: Multiple hypotheses evaluated in sequence, each processed as if passed individually. The resulting rows are stacked in the order supplied. Example: `["b1 - b0 = 0", "b2 = 1"]`.
- numpy.ndarray: Each column is a vector of weights. The output is the dot product between these vectors of weights and the vectors of estimates. e.g. `hypothesis=np.array([[1, 1, 2], [2, 2, 3]]).T`
- See the Examples section and the vignette: https://marginaleffects.com/chapters/hypothesis.html
`wts`: (str, optional) Column name of weights to use for marginalization. Must be a column in `newdata`.
`vcov`: (bool, np.ndarray, default=True) Type of uncertainty estimates to report (e.g. for robust standard errors). Acceptable values are:
- `True`: Use the model's default covariance matrix.
- `False`: Do not compute standard errors.
- String: Literal indicating the kind of uncertainty estimates to return:
- Heteroskedasticity-consistent: `"HC0"`, `"HC1"`, `"HC2"`, `"HC3"`.
- np.ndarray: A custom square covariance matrix.
`equivalence`: (list, optional) List of 2 numeric float values specifying the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. See the Details section below.
`cross`: `False` Contrasts represent the change in adjusted predictions when one predictor changes and all other variables are held constant. `True` Contrasts represent the changes in adjusted predictions when all the predictors specified in the variables argument are manipulated simultaneously (a "cross-contrast").
`conf_level`: (float, default=0.95) Numeric value specifying the confidence level for the confidence intervals.
`eps`: (float, optional) step size to use when calculating numerical derivatives: (f(x+eps)-f(x))/eps. Default value is 1e-4 multiplied by the difference between the maximum and minimum values of the variable with respect to which we are taking the derivative. Changing eps may be necessary to avoid numerical problems in certain models.
`eps_vcov`: (float) optional custom value for the finite difference approximation of the jacobian matrix. By default, the function uses the square root of the machine epsilon.
## Returns {.unnumbered}
A Polars DataFrame with (some of) the following columns:
- `term`: the name of the variable.
- `contrast`: the comparison method used.
- `estimate`: the estimated contrast, difference, ratio, or other transformation between pairs of predictions.
- `std_error`: the standard error of the estimate.
- `statistic`: the test statistic (estimate / std.error).
- `p_value`: the p-value of the test.
- `s_value`: Shannon transform of the p value.
- `conf_low`: the lower confidence interval bound.
- `conf_high`: the upper confidence interval bound.
- `pred_low`: the lower prediction interval bound.
- `pred_high`: the upper prediction interval bound.
## Examples {.unnumbered}
```py
from marginaleffects import *
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf
data = get_dataset("thornton")
model = smf.ols("outcome ~ distance + incentive", data=data).fit()
# Basic comparisons {.unnumbered}
comparisons(model)
avg_comparisons(model)
comparisons(model, hypothesis=0)
avg_comparisons(model, hypothesis=0)
comparisons(model, by="agecat")
avg_comparisons(model, by="agecat")
# Custom comparisons with lambda functions {.unnumbered}
# Ratio comparison using lambda {.unnumbered}
comparisons(model, variables="distance",
comparison=lambda hi, lo, eps, x, y, w: hi / lo)
# Percent change using lambda {.unnumbered}
comparisons(model, variables="distance",
comparison=lambda hi, lo, eps, x, y, w: (hi - lo) / lo * 100)
# Log ratio using lambda {.unnumbered}
comparisons(model, variables="distance",
comparison=lambda hi, lo, eps, x, y, w: np.log(hi / lo))
```
## Details {.unnumbered}
### Two-One-Sided Test (TOST) of Equivalence {.unnumbered}
The `equivalence` argument specifies the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. The first element specifies the lower bound, and the second element specifies the upper bound. If `None`, equivalence tests are not performed.
### Order of operations. {.unnumbered}
Behind the scenes, the arguments of `marginaleffects` functions are evaluated in this order:
1. `newdata`
2. `variables`
3. `comparison` and `slope`
4. `by`
5. `vcov`
6. `hypothesis`
7. `transform`
---
# `comparisons()` {.unnumbered}
`comparisons()` and `avg_comparisons()` are functions for predicting the outcome variable at different regressor values and comparing those predictions by computing a difference, ratio, or some other function. These functions can return many quantities of interest, such as contrasts, differences, risk ratios, changes in log odds, lift, slopes, elasticities, average treatment effect (on the treated or untreated), etc.
* `comparisons()`: unit-level (conditional) estimates.
* `avg_comparisons()`: average (marginal) estimates.
See the package website and vignette for examples:
* https://marginaleffects.com/chapters/comparisons.html
* https://marginaleffects.com
## Parameters {.unnumbered}
`model`: (model object) Object fitted using the `statsmodels` formula API.
`variables`: (str, list, dictionary) Specifies what variables (columns) to vary in order to make the comparison.
- `None`: comparisons are computed for all regressors in the model object (can be slow). Acceptable values depend on the variable type. See the examples below.
- List[str] or str: List of variable names to compute comparisons for.
- Dictionary: keys identify the subset of variables of interest, and values define the type of contrast to compute. Acceptable values depend on the variable type:
- Categorical variables:
* "reference": Each factor level is compared to the factor reference (base) level
* "all": All combinations of observed levels
* "sequential": Each factor level is compared to the previous factor level
* "pairwise": Each factor level is compared to all other levels
* "minmax": The highest and lowest levels of a factor.
* "revpairwise", "revreference", "revsequential": inverse of the corresponding hypotheses.
* Vector of length 2 with the two values to compare.
- Boolean variables:
* `None`: contrast between True and False
- Numeric variables:
* Numeric of length 1: Contrast for a gap of `x`, computed at the observed value plus and minus `x / 2`. For example, estimating a `+1` contrast compares adjusted predictions when the regressor is equal to its observed value minus 0.5 and its observed value plus 0.5.
* Numeric of length equal to the number of rows in `newdata`: Same as above, but the contrast can be customized for each row of `newdata`.
* Numeric vector of length 2: Contrast between the 2nd element and the 1st element of the `x` vector.
* Data frame with the same number of rows as `newdata`, with two columns of "low" and "high" values to compare.
* Function which accepts a numeric vector and returns a data frame with two columns of "low" and "high" values to compare. See examples below.
* "iqr": Contrast across the interquartile range of the regressor.
* "sd": Contrast across one standard deviation around the regressor mean.
* "2sd": Contrast across two standard deviations around the regressor mean.
* "minmax": Contrast between the maximum and the minimum values of the regressor.
- Examples:
+ `variables = "gear" : "pairwise", "hp" : 10`
+ `variables = "gear" : "sequential", "hp" : [100, 120]`
`newdata`: (None, DataFrame, str) Data frame or string specifying where statistics are evaluated in the predictor space.
- None: Compute comparisons at each observed value in the original dataset (empirical distribution)
- Dataframe: should be created with datagrid() function
- str:
* "mean": Compute comparisons at the mean of the regressor
* "median": Compute comparisons at the median of the regressor
* "balanced": Compute comparisons on a balanced grid with every combination of categories and numeric variables held at their means.
* "tukey": Probably NotImplemented
* "grid": Probably NotImplemented
* `comparison`: (str or callable) String specifying how pairs of predictions should be compared, or a callable function to compute custom estimates. See the Comparisons section below for definitions of each transformation.
* Acceptable strings: difference, differenceavg, differenceavgwts, dydx, eyex, eydx, dyex, dydxavg, eyexavg, eydxavg, dyexavg, dydxavgwts, eyexavgwts, eydxavgwts, dyexavgwts, ratio, ratioavg, ratioavgwts, lnratio, lnratioavg, lnratioavgwts, lnor, lnoravg, lnoravgwts, lift, liftavg, liftavg, expdydx, expdydxavg, expdydxavgwts
* Callable: A function that takes `hi`, `lo`, `eps`, `x`, `y`, and `w` as arguments and returns a numeric array. This allows computing custom comparisons like `lambda hi, lo, eps, x, y, w: hi / lo` for ratios or `lambda hi, lo, eps, x, y, w: (hi - lo) / lo * 100` for percent changes.
`by`: (bool, List[str], optional) A logical value or a list of column names in `newdata`.
- `True`: estimate is aggregated across the whole dataset.
- list: estimates are aggregated for each unique combination of values in the columns.
`transform`: (function) Function specifying a transformation applied to unit-level estimates and confidence intervals just before the function returns results. Functions must accept a full column (series) of a Polars data frame and return a corresponding series of the same length. Ex:
- `transform = numpy.exp`
- `transform = lambda x: x.exp()`
- `transform = lambda x: x.map_elements()`
`hypothesis`: (str, int, float, list of str, numpy array) Specifies a hypothesis test or custom contrast
* Number to specify the null hypothesis.
* Numpy array with a number of rows equal to the number of estimates.
* String equation with an equal sign and estimate number in b0, b1, b2, etc. format.
- "b0 = b1"
- "b0 - (b1 + b2) = 0"
* Two-side formula like "ratio ~ reference"
- Left-hand side: "ratio", "difference"
- Right-hand side: 'reference', 'sequential', 'pairwise', 'revreference', 'revsequential', 'revpairwise'
- int, float: The null hypothesis used in the computation of Z and p-values (before applying transform)
- str:
* equation specifying linear or non-linear hypothesis tests. Use the names of the model variables, or use `b0`, `b1` to identify the position of each parameter. The `b*` wildcard can be used to test hypotheses on all estimates. Examples:
- `hp = drat`
- `hp + drat = 12`
- `b0 + b1 + b2 = 0`
- `b* / b0 = 1`
* one of the following hypothesis test strings:
- `pairwise` and `revpairwise`: pairwise differences between estimates in each row.
- `reference` and `revreference`: differences between the estimates in each row and the estimate in the first row.
- `sequential` and `revsequential`: differences between an estimate and the estimate in the next row.
- list of strings: Multiple hypotheses evaluated in sequence, each processed as if passed individually. The resulting rows are stacked in the order supplied. Example: `["b1 - b0 = 0", "b2 = 1"]`.
- numpy.ndarray: Each column is a vector of weights. The output is the dot product between these vectors of weights and the vectors of estimates. e.g. `hypothesis=np.array([[1, 1, 2], [2, 2, 3]]).T`
- See the Examples section and the vignette: https://marginaleffects.com/chapters/hypothesis.html
`wts`: (str, optional) Column name of weights to use for marginalization. Must be a column in `newdata`.
`vcov`: (bool, np.ndarray, default=True) Type of uncertainty estimates to report (e.g. for robust standard errors). Acceptable values are:
- `True`: Use the model's default covariance matrix.
- `False`: Do not compute standard errors.
- String: Literal indicating the kind of uncertainty estimates to return:
- Heteroskedasticity-consistent: `"HC0"`, `"HC1"`, `"HC2"`, `"HC3"`.
- np.ndarray: A custom square covariance matrix.
`equivalence`: (list, optional) List of 2 numeric float values specifying the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. See the Details section below.
`cross`: `False` Contrasts represent the change in adjusted predictions when one predictor changes and all other variables are held constant. `True` Contrasts represent the changes in adjusted predictions when all the predictors specified in the variables argument are manipulated simultaneously (a "cross-contrast").
`conf_level`: (float, default=0.95) Numeric value specifying the confidence level for the confidence intervals.
`eps`: (float, optional) step size to use when calculating numerical derivatives: (f(x+eps)-f(x))/eps. Default value is 1e-4 multiplied by the difference between the maximum and minimum values of the variable with respect to which we are taking the derivative. Changing eps may be necessary to avoid numerical problems in certain models.
`eps_vcov`: (float) optional custom value for the finite difference approximation of the jacobian matrix. By default, the function uses the square root of the machine epsilon.
## Returns {.unnumbered}
A Polars DataFrame with (some of) the following columns:
- `term`: the name of the variable.
- `contrast`: the comparison method used.
- `estimate`: the estimated contrast, difference, ratio, or other transformation between pairs of predictions.
- `std_error`: the standard error of the estimate.
- `statistic`: the test statistic (estimate / std.error).
- `p_value`: the p-value of the test.
- `s_value`: Shannon transform of the p value.
- `conf_low`: the lower confidence interval bound.
- `conf_high`: the upper confidence interval bound.
- `pred_low`: the lower prediction interval bound.
- `pred_high`: the upper prediction interval bound.
## Examples {.unnumbered}
```py
from marginaleffects import *
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf
data = get_dataset("thornton")
model = smf.ols("outcome ~ distance + incentive", data=data).fit()
# Basic comparisons {.unnumbered}
comparisons(model)
avg_comparisons(model)
comparisons(model, hypothesis=0)
avg_comparisons(model, hypothesis=0)
comparisons(model, by="agecat")
avg_comparisons(model, by="agecat")
# Custom comparisons with lambda functions {.unnumbered}
# Ratio comparison using lambda {.unnumbered}
comparisons(model, variables="distance",
comparison=lambda hi, lo, eps, x, y, w: hi / lo)
# Percent change using lambda {.unnumbered}
comparisons(model, variables="distance",
comparison=lambda hi, lo, eps, x, y, w: (hi - lo) / lo * 100)
# Log ratio using lambda {.unnumbered}
comparisons(model, variables="distance",
comparison=lambda hi, lo, eps, x, y, w: np.log(hi / lo))
```
## Details {.unnumbered}
### Two-One-Sided Test (TOST) of Equivalence {.unnumbered}
The `equivalence` argument specifies the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. The first element specifies the lower bound, and the second element specifies the upper bound. If `None`, equivalence tests are not performed.
### Order of operations. {.unnumbered}
Behind the scenes, the arguments of `marginaleffects` functions are evaluated in this order:
1. `newdata`
2. `variables`
3. `comparison` and `slope`
4. `by`
5. `vcov`
6. `hypothesis`
7. `transform`
---
# `slopes()` {.unnumbered}
`slopes()` and `avg_slopes()` estimate unit-level (conditional) partial derivative of the regression equation with respect to a regressor of interest.
* `slopes()`: unit-level (conditional) estimates.
* `avg_slopes()`: average (marginal) estimates.
The newdata argument and the `datagrid()` function can be used to control where statistics are evaluated in the predictor space: "at observed values", "at the mean", "at representative values", etc.
See the package website and vignette for examples:
- https://marginaleffects.com/chapters/slopes.html
- https://marginaleffects.com
## Parameters {.unnumbered}
`model`: (model object) Object fitted using the `statsmodels` formula API.
`variables`: (str, list, dictionary) Specifies what variables (columns) to vary in order to make the slopes.
- str: Variable for which to compute the slopes for.
- NoneType: Slopes are computed for all regressors in the model object (can be slow)
`newdata`: (None, DataFrame, str) Data frame or string specifying where statistics are evaluated in the predictor space.
- None: Compute slopes at each observed value in the original dataset (empirical distribution)
- Dataframe: should be created with datagrid() function
- str:
* "mean": Compute slopes at the mean of the regressor
* "median": Compute slopes at the median of the regressor
* "balanced": Compute slopes on a balanced grid with every combination of categories and numeric variables held at their means.
* "tukey": Probably NotImplemented
* "grid": Probably NotImplemented
`slope`: (str) The type of slope or (semi-)elasticity to compute. Acceptable values are:
- "dydx": dY/dX
- "eyex": dY/dX * Y / X
- "eydx": dY/dX * Y
- "dyex": dY/dX / X
`vcov`: (bool, np.ndarray, default=True) Type of uncertainty estimates to report (e.g. for robust standard errors). Acceptable values are:
- `True`: Use the model's default covariance matrix.
- `False`: Do not compute standard errors.
- String: Literal indicating the kind of uncertainty estimates to return:
- Heteroskedasticity-consistent: `"HC0"`, `"HC1"`, `"HC2"`, `"HC3"`.
- np.ndarray: A custom square covariance matrix.
`conf_level`: (float, default=0.95) Numeric value specifying the confidence level for the confidence intervals.
`by`: (bool, List[str], optional) A logical value or a list of column names in `newdata`.
- `True`: estimate is aggregated across the whole dataset.
- list: estimates are aggregated for each unique combination of values in the columns.
`hypothesis`: (str, int, float, list of str, numpy array) Specifies a hypothesis test or custom contrast
* Number to specify the null hypothesis.
* Numpy array with a number of rows equal to the number of estimates.
* String equation with an equal sign and estimate number in b0, b1, b2, etc. format.
- "b0 = b1"
- "b0 - (b1 + b2) = 0"
* Two-side formula like "ratio ~ reference"
- Left-hand side: "ratio", "difference"
- Right-hand side: 'reference', 'sequential', 'pairwise', 'revreference', 'revsequential', 'revpairwise'
- int, float: The null hypothesis used in the computation of Z and p-values (before applying transform)
- str:
* equation specifying linear or non-linear hypothesis tests. Use the names of the model variables, or use `b0`, `b1` to identify the position of each parameter. The `b*` wildcard can be used to test hypotheses on all estimates. Examples:
- `hp = drat`
- `hp + drat = 12`
- `b0 + b1 + b2 = 0`
- `b* / b0 = 1`
* one of the following hypothesis test strings:
- `pairwise` and `revpairwise`: pairwise differences between estimates in each row.
- `reference` and `revreference`: differences between the estimates in each row and the estimate in the first row.
- `sequential` and `revsequential`: differences between an estimate and the estimate in the next row.
- list of strings: Multiple hypotheses evaluated in sequence, each processed as if passed individually. The resulting rows are stacked in the order supplied. Example: `["b1 - b0 = 0", "b2 = 1"]`.
- numpy.ndarray: Each column is a vector of weights. The output is the dot product between these vectors of weights and the vectors of estimates. e.g. `hypothesis=np.array([[1, 1, 2], [2, 2, 3]]).T`
- See the Examples section and the vignette: https://marginaleffects.com/chapters/hypothesis.html
`equivalence`: (list, optional) List of 2 numeric float values specifying the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. See the Details section below.
`wts`: (str, optional) Column name of weights to use for marginalization. Must be a column in `newdata`.
`eps`: (float, optional) step size to use when calculating numerical derivatives: (f(x+eps)-f(x))/eps. Default value is 1e-4 multiplied by the difference between the maximum and minimum values of the variable with respect to which we are taking the derivative. Changing eps may be necessary to avoid numerical problems in certain models.
`eps_vcov`: (float) optional custom value for the finite difference approximation of the jacobian matrix. By default, the function uses the square root of the machine epsilon.
## Returns {.unnumbered}
A Polars DataFrame with (some of) the following columns:
- `term`: the name of the variable.
- `contrast`: the comparison method used.
- `estimate`: the estimated contrast, difference, ratio, or other transformation between pairs of predictions.
- `std_error`: the standard error of the estimate.
- `statistic`: the test statistic (estimate / std.error).
- `p_value`: the p-value of the test.
- `s_value`: Shannon transform of the p value.
- `conf_low`: the lower confidence interval bound.
- `conf_high`: the upper confidence interval bound.
- `pred_low`: the lower prediction interval bound.
- `pred_high`: the upper prediction interval bound.
## Examples {.unnumbered}
```py
from marginaleffects import *
import statsmodels.api as sm
import statsmodels.formula.api as smf
import numpy as np
data = get_dataset("thornton")
# Create the squared distance term {.unnumbered}
data = data.with_columns(distance_sq = data['distance'] ** 2)
# Fit GLM with interactions and squared term {.unnumbered}
mod = smf.logit("outcome ~ incentive * distance * distance_sq", data=data).fit()
# Print summary {.unnumbered}
mod.summary()
# Slopes are computed for each regressor and at each observation {.unnumbered}
slopes(mod)
# Slopes are computed for each regressor and averaged over the observations {.unnumbered}
avg_slopes(mod)
# Slopes are computed with respect to the `distance` regressor at each observation {.unnumbered}
slopes(mod, variables = "distance")
# Slopes are computed with respect to the `distance` regressor and averaged over the observations {.unnumbered}
avg_slopes(mod, variables = "distance")
```
## Details {.unnumbered}
### Two-One-Sided Test (TOST) of Equivalence {.unnumbered}
The `equivalence` argument specifies the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. The first element specifies the lower bound, and the second element specifies the upper bound. If `None`, equivalence tests are not performed.
### Order of operations. {.unnumbered}
Behind the scenes, the arguments of `marginaleffects` functions are evaluated in this order:
1. `newdata`
2. `variables`
3. `comparison` and `slope`
4. `by`
5. `vcov`
6. `hypothesis`
7. `transform`
---
# `slopes()` {.unnumbered}
`slopes()` and `avg_slopes()` estimate unit-level (conditional) partial derivative of the regression equation with respect to a regressor of interest.
* `slopes()`: unit-level (conditional) estimates.
* `avg_slopes()`: average (marginal) estimates.
The newdata argument and the `datagrid()` function can be used to control where statistics are evaluated in the predictor space: "at observed values", "at the mean", "at representative values", etc.
See the package website and vignette for examples:
- https://marginaleffects.com/chapters/slopes.html
- https://marginaleffects.com
## Parameters {.unnumbered}
`model`: (model object) Object fitted using the `statsmodels` formula API.
`variables`: (str, list, dictionary) Specifies what variables (columns) to vary in order to make the slopes.
- str: Variable for which to compute the slopes for.
- NoneType: Slopes are computed for all regressors in the model object (can be slow)
`newdata`: (None, DataFrame, str) Data frame or string specifying where statistics are evaluated in the predictor space.
- None: Compute slopes at each observed value in the original dataset (empirical distribution)
- Dataframe: should be created with datagrid() function
- str:
* "mean": Compute slopes at the mean of the regressor
* "median": Compute slopes at the median of the regressor
* "balanced": Compute slopes on a balanced grid with every combination of categories and numeric variables held at their means.
* "tukey": Probably NotImplemented
* "grid": Probably NotImplemented
`slope`: (str) The type of slope or (semi-)elasticity to compute. Acceptable values are:
- "dydx": dY/dX
- "eyex": dY/dX * Y / X
- "eydx": dY/dX * Y
- "dyex": dY/dX / X
`vcov`: (bool, np.ndarray, default=True) Type of uncertainty estimates to report (e.g. for robust standard errors). Acceptable values are:
- `True`: Use the model's default covariance matrix.
- `False`: Do not compute standard errors.
- String: Literal indicating the kind of uncertainty estimates to return:
- Heteroskedasticity-consistent: `"HC0"`, `"HC1"`, `"HC2"`, `"HC3"`.
- np.ndarray: A custom square covariance matrix.
`conf_level`: (float, default=0.95) Numeric value specifying the confidence level for the confidence intervals.
`by`: (bool, List[str], optional) A logical value or a list of column names in `newdata`.
- `True`: estimate is aggregated across the whole dataset.
- list: estimates are aggregated for each unique combination of values in the columns.
`hypothesis`: (str, int, float, list of str, numpy array) Specifies a hypothesis test or custom contrast
* Number to specify the null hypothesis.
* Numpy array with a number of rows equal to the number of estimates.
* String equation with an equal sign and estimate number in b0, b1, b2, etc. format.
- "b0 = b1"
- "b0 - (b1 + b2) = 0"
* Two-side formula like "ratio ~ reference"
- Left-hand side: "ratio", "difference"
- Right-hand side: 'reference', 'sequential', 'pairwise', 'revreference', 'revsequential', 'revpairwise'
- int, float: The null hypothesis used in the computation of Z and p-values (before applying transform)
- str:
* equation specifying linear or non-linear hypothesis tests. Use the names of the model variables, or use `b0`, `b1` to identify the position of each parameter. The `b*` wildcard can be used to test hypotheses on all estimates. Examples:
- `hp = drat`
- `hp + drat = 12`
- `b0 + b1 + b2 = 0`
- `b* / b0 = 1`
* one of the following hypothesis test strings:
- `pairwise` and `revpairwise`: pairwise differences between estimates in each row.
- `reference` and `revreference`: differences between the estimates in each row and the estimate in the first row.
- `sequential` and `revsequential`: differences between an estimate and the estimate in the next row.
- list of strings: Multiple hypotheses evaluated in sequence, each processed as if passed individually. The resulting rows are stacked in the order supplied. Example: `["b1 - b0 = 0", "b2 = 1"]`.
- numpy.ndarray: Each column is a vector of weights. The output is the dot product between these vectors of weights and the vectors of estimates. e.g. `hypothesis=np.array([[1, 1, 2], [2, 2, 3]]).T`
- See the Examples section and the vignette: https://marginaleffects.com/chapters/hypothesis.html
`equivalence`: (list, optional) List of 2 numeric float values specifying the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. See the Details section below.
`wts`: (str, optional) Column name of weights to use for marginalization. Must be a column in `newdata`.
`eps`: (float, optional) step size to use when calculating numerical derivatives: (f(x+eps)-f(x))/eps. Default value is 1e-4 multiplied by the difference between the maximum and minimum values of the variable with respect to which we are taking the derivative. Changing eps may be necessary to avoid numerical problems in certain models.
`eps_vcov`: (float) optional custom value for the finite difference approximation of the jacobian matrix. By default, the function uses the square root of the machine epsilon.
## Returns {.unnumbered}
A Polars DataFrame with (some of) the following columns:
- `term`: the name of the variable.
- `contrast`: the comparison method used.
- `estimate`: the estimated contrast, difference, ratio, or other transformation between pairs of predictions.
- `std_error`: the standard error of the estimate.
- `statistic`: the test statistic (estimate / std.error).
- `p_value`: the p-value of the test.
- `s_value`: Shannon transform of the p value.
- `conf_low`: the lower confidence interval bound.
- `conf_high`: the upper confidence interval bound.
- `pred_low`: the lower prediction interval bound.
- `pred_high`: the upper prediction interval bound.
## Examples {.unnumbered}
```py
from marginaleffects import *
import statsmodels.api as sm
import statsmodels.formula.api as smf
import numpy as np
data = get_dataset("thornton")
# Create the squared distance term {.unnumbered}
data = data.with_columns(distance_sq = data['distance'] ** 2)
# Fit GLM with interactions and squared term {.unnumbered}
mod = smf.logit("outcome ~ incentive * distance * distance_sq", data=data).fit()
# Print summary {.unnumbered}
mod.summary()
# Slopes are computed for each regressor and at each observation {.unnumbered}
slopes(mod)
# Slopes are computed for each regressor and averaged over the observations {.unnumbered}
avg_slopes(mod)
# Slopes are computed with respect to the `distance` regressor at each observation {.unnumbered}
slopes(mod, variables = "distance")
# Slopes are computed with respect to the `distance` regressor and averaged over the observations {.unnumbered}
avg_slopes(mod, variables = "distance")
```
## Details {.unnumbered}
### Two-One-Sided Test (TOST) of Equivalence {.unnumbered}
The `equivalence` argument specifies the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. The first element specifies the lower bound, and the second element specifies the upper bound. If `None`, equivalence tests are not performed.
### Order of operations. {.unnumbered}
Behind the scenes, the arguments of `marginaleffects` functions are evaluated in this order:
1. `newdata`
2. `variables`
3. `comparison` and `slope`
4. `by`
5. `vcov`
6. `hypothesis`
7. `transform`
---
# `hypotheses()` {.unnumbered}
(Non-)Linear Tests for Null Hypotheses, Joint Hypotheses, Equivalence, Non Superiority, and Non Inferiority.
This function calculates uncertainty estimates as first-order approximate standard errors for linear or non-linear
functions of a vector of random variables with known or estimated covariance matrix. It emulates the behavior of
the excellent and well-established `car::deltaMethod` and `car::linearHypothesis` functions in R, but it supports
more models; requires fewer dependencies; expands the range of tests to equivalence and superiority/inferiority;
and offers convenience features like robust standard errors.
To learn more, visit the package website:
## Parameters {.unnumbered}
* model : object
Model object estimated by `statsmodels`
`hypothesis`: (str, int, float, list of str, numpy array) Specifies a hypothesis test or custom contrast
* Number to specify the null hypothesis.
* Numpy array with a number of rows equal to the number of estimates.
* String equation with an equal sign and estimate number in b0, b1, b2, etc. format.
- "b0 = b1"
- "b0 - (b1 + b2) = 0"
* Two-side formula like "ratio ~ reference"
- Left-hand side: "ratio", "difference"
- Right-hand side: 'reference', 'sequential', 'pairwise', 'revreference', 'revsequential', 'revpairwise'
- int, float: The null hypothesis used in the computation of Z and p-values (before applying transform)
- str:
* equation specifying linear or non-linear hypothesis tests. Use the names of the model variables, or use `b0`, `b1` to identify the position of each parameter. The `b*` wildcard can be used to test hypotheses on all estimates. Examples:
- `hp = drat`
- `hp + drat = 12`
- `b0 + b1 + b2 = 0`
- `b* / b0 = 1`
* one of the following hypothesis test strings:
- `pairwise` and `revpairwise`: pairwise differences between estimates in each row.
- `reference` and `revreference`: differences between the estimates in each row and the estimate in the first row.
- `sequential` and `revsequential`: differences between an estimate and the estimate in the next row.
- list of strings: Multiple hypotheses evaluated in sequence, each processed as if passed individually. The resulting rows are stacked in the order supplied. Example: `["b1 - b0 = 0", "b2 = 1"]`.
- numpy.ndarray: Each column is a vector of weights. The output is the dot product between these vectors of weights and the vectors of estimates. e.g. `hypothesis=np.array([[1, 1, 2], [2, 2, 3]]).T`
- See the Examples section and the vignette: https://marginaleffects.com/chapters/hypothesis.html
`conf_level`: (float, default=0.95) Numeric value specifying the confidence level for the confidence intervals.
`vcov`: (bool, np.ndarray, default=True) Type of uncertainty estimates to report (e.g. for robust standard errors). Acceptable values are:
- `True`: Use the model's default covariance matrix.
- `False`: Do not compute standard errors.
- String: Literal indicating the kind of uncertainty estimates to return:
- Heteroskedasticity-consistent: `"HC0"`, `"HC1"`, `"HC2"`, `"HC3"`.
- np.ndarray: A custom square covariance matrix.
`equivalence`: (list, optional) List of 2 numeric float values specifying the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. See the Details section below.
`eps_vcov`: (float) optional custom value for the finite difference approximation of the jacobian matrix. By default, the function uses the square root of the machine epsilon.
* joint: (bool, str, List[str], default = `False`) Specifies the joint test of statistical significance. The null hypothesis value can be set using the hypothesis argument.
- `False`: Hypothesis are not tested jointly
- `True`: Hypothesis are tested jointly
- str: A regular expression to match parameters to be tested jointly.
- List[str]: Parameter names to be tested jointly as displayed by `mod.model.data.param_names`
- List[int]: Parameter positions to test jointly where positions refer to the order specified by `mod.model.data.param_names`
* joint_test: (str, default=`"f"`) Chooses the type of test between `"f"` and `"chisq"`
## Returns {.unnumbered}
(MarginaleffectsResult)
* DataFrame containing the results of the hypothesis tests.
## Examples {.unnumbered}
```py
from marginaleffects import *
import statsmodels.api as sm
import statsmodels.formula.api as smf
data = get_dataset("thornton")
model = smf.ols("outcome ~ distance + incentive", data=data).fit()
# When `hypothesis` is `None`, `hypotheses()` returns a DataFrame of parameters {.unnumbered}
hypotheses(model)
# A different null hypothesis {.unnumbered}
hypotheses(model, hypothesis = 3)
# Test of equality between coefficients {.unnumbered}
hypotheses(model, hypothesis="distance = incentive")
# Non-linear function {.unnumbered}
hypotheses(model, hypothesis="(distance + incentive) = 0.1")
# Robust standard errors {.unnumbered}
hypotheses(model, hypothesis="distance = incentive", vcov="HC3")
# Equivalence, non-inferiority, and non-superiority tests {.unnumbered}
hypotheses(model, equivalence=(0.0, 10.0))
# Joint hypothesis tests {.unnumbered}
hypotheses(model, joint=["distance", "incentive"])
# Joint hypothesis tests with a regular expression {.unnumbered}
hypotheses(model, joint="distance|incentive")
# Joint hypothesis tests with a regular expression {.unnumbered}
hypotheses(model, joint="i$") # matches `incentive` and `distance` columns
```
## Warnings {.unnumbered}
* Warning #1: Tests are conducted directly on the scale defined by the `type` argument. For some models, it can make sense to conduct hypothesis or equivalence tests on the `"link"` scale instead of the `"response"` scale which is often the default.
* Warning #2: For hypothesis tests on objects produced by the `marginaleffects` package, it is safer to use the `hypothesis` argument of the original function.
* Warning #3: The tests assume that the `hypothesis` expression is (approximately) normally distributed, which for non-linear functions of the parameters may not be realistic. More reliable confidence intervals can be obtained using the `inferences()` (in R only) function with `method = "boot"`
## Details {.unnumbered}
### Two-One-Sided Test (TOST) of Equivalence {.unnumbered}
The `equivalence` argument specifies the bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. The first element specifies the lower bound, and the second element specifies the upper bound. If `None`, equivalence tests are not performed.
---
# `datagrid()` {.unnumbered}
Generate a data grid of user-specified values for use in the 'newdata' argument of the 'predictions()', 'comparisons()', and 'slopes()' functions.
This is useful to define where in the predictor space we want to evaluate the quantities of interest. Ex: the predicted outcome or slope for a 37 year old college graduate.
## Parameters {.unnumbered}
* model: (object, optional)
Model object.
* (one and only one of the `model` and `newdata` arguments can be used.)
* newdata: (DataFrame, optional)
Data frame used to define the predictor space.
* (one and only one of the `model` and `newdata` arguments can be used.)
* grid_type: (str, optional)
Determines the functions to apply to each variable. The defaults can be overridden by defining individual variables explicitly in the `**kwargs`, or by supplying a function to one of the `FUN_*` arguments.
* "mean_or_mode": Character, factor, logical, and binary variables are set to their modes. Numeric, integer, and other variables are set to their means.
* "balanced": Each unique level of character, factor, logical, and binary variables are preserved. Numeric, integer, and other variables are set to their means. Warning: When there are many variables and many levels per variable, a balanced grid can be very large. In those cases, it is better to use `grid_type="mean_or_mode"` and to specify the unique levels of a subset of named variables explicitly.
* "counterfactual": the entire dataset is duplicated for each combination of the variable values specified in `**kwargs`. Variables not explicitly supplied to `datagrid()` are set to their observed values in the original dataset.
* FUN_numeric: (Callable, optional)
The function to be applied to numeric variables.
* FUN_other: (Callable, optional)
The function to be applied to other variable types.
* **kwargs
* Named arguments where the name is the variable name and the value is a list of values to use in the grid. If a variable is not specified, it is set to its mean or mode depending on the `grid_type` argument.
## Returns {.unnumbered}
(polars.DataFrame)
* DataFrame where each row corresponds to one combination of the named predictors supplied by the user. Variables which are not explicitly defined are held at their mean or mode.
## Examples {.unnumbered}
```py
import polars as pl
import statsmodels.formula.api as smf
from marginaleffects import *
data = get_dataset("thornton")
# The output only has 2 rows, and all the variables except `hp` are at their mean or mode. {.unnumbered}
datagrid(newdata = data, village = [43, 11])
# We get the same result by feeding a model instead of a DataFrame {.unnumbered}
mod = smf.ols("outcome ~ incentive + distance", data).fit()
datagrid(model = mod, village = [43, 11])
# Use in `marginaleffects` to compute "Typical Marginal Effects". When used in `slopes()` or `predictions()` we do not need to specify the `model` or `newdata` arguments. {.unnumbered}
nd = datagrid(mod, village = [43, 11])
slopes(mod, newdata = nd)
# The full dataset is duplicated with each observation given counterfactual values of 43 and 11 for the `village` variable. {.unnumbered}
# The original `thornton` includes 2884 rows, so the resulting dataset includes 5768 rows. {.unnumbered}
dg = datagrid(newdata = data, village = [43, 11], grid_type = "counterfactual")
dg.shape
```
---
# `plot_predictions()` {.unnumbered}
Plot predictions on the y-axis against values of one or more predictors (x-axis, colors/shapes, and facets).
The `by` argument is used to plot marginal predictions, that is, predictions made on the original data, but averaged
by subgroups. This is analogous to using the `by` argument in the `predictions()` function.
The `condition` argument is used to plot conditional predictions, that is, predictions made on a user-specified grid.
This is analogous to using the `newdata` argument and `datagrid()` function in a `predictions()` call.
All unspecified variables are held at their mean or mode. This includes grouping variables in mixed-effects models, so analysts who fit such models may want to specify the groups of interest using the `variables` argument, or supply model-specific arguments to compute population-level estimates. See details below.
See the "Plots" vignette and website for tutorials and information on how to customize plots:
- https://marginaleffects.com/articles/plot.html
- https://marginaleffects.com
## Parameters {.unnumbered}
`model`: (model object) Object fitted using the `statsmodels` formula API.
`condition`: (str, list, dictionary) Conditional predictionss.
- Position's representation:
1. x-axis.
2. color.
3. facet (wrap if no fourth variable, otherwise cols of grid).
4. facet (rows of grid).
- Argument types:
- list : Names of the predictors to display
- Numeric variables in position 1 is summarized by 100 numbers
- Numeric variables in positions 2, 3 and 4 are summarized by Tukey’s five numbers
- dictionary : Keys correspond to predictors. Values can be one of the two below depending on predictor's type:
- Series or list of the same type as the original variable.
- Numeric variables:
- String: "minmax", "threenum", "fivenum".
- string : Same as list of length 1.
`by`: (bool, str, list) Marginal predictionss.
Names of the categorical predictors to marginalize across. Max length of list is 4, with position meanings:
1. x-axis.
2. color.
3. facet (wrap if no fourth variable, otherwise columns of grid).
4. facet (rows of grid)
`draw`: True returns a matplotlib plot. False returns a dataframe of the underlying data.
`newdata`: (dataframe) When newdata is `None`, the grid is determined by the condition argument. When newdata is not `None`, the argument behaves in the same way as in the predictions() function.
`vcov`: (bool, np.ndarray, default=True) Type of uncertainty estimates to report (e.g. for robust standard errors). Acceptable values are:
- `True`: Use the model's default covariance matrix.
- `False`: Do not compute standard errors.
- String: Literal indicating the kind of uncertainty estimates to return:
- Heteroskedasticity-consistent: `"HC0"`, `"HC1"`, `"HC2"`, `"HC3"`.
- np.ndarray: A custom square covariance matrix.
`wts`: (str, optional) Column name of weights to use for marginalization. Must be a column in `newdata`.
`transform`: (function) Function specifying a transformation applied to unit-level estimates and confidence intervals just before the function returns results. Functions must accept a full column (series) of a Polars data frame and return a corresponding series of the same length. Ex:
- `transform = numpy.exp`
- `transform = lambda x: x.exp()`
- `transform = lambda x: x.map_elements()`
`points`: (float, default=0) Number between 0 and 1 which controls the transparency of raw data points. 0 (default) does not display any points.
Warning: The points displayed are raw data, so the resulting plot is not a "partial residual plot."
`gray`: True returns a gray scale adapted plot. False returns a plot in color. For the second position of the list in the `condition` or `by` argument, the list can have at most 5 elements.
## Examples {.unnumbered}
```py
from marginaleffects import *
import statsmodels.api as sm
import statsmodels.formula.api as smf
data = get_dataset("thornton")
mod = smf.ols("outcome ~ incentive + distance", data).fit()
plot_predictions(mod, condition = ["distance", "incentive"])
```
---
# `plot_comparisons()` {.unnumbered}
Plot comparisons on the y-axis against values of one or more predictors (x-axis, colors/shapes, and facets).
The `by` argument is used to plot marginal comparisons, that is, comparisons made on the original data, but averaged
by subgroups. This is analogous to using the `by` argument in the `comparisons()` function.
The `condition` argument is used to plot conditional comparisons, that is, comparisons made on a user-specified grid.
This is analogous to using the `newdata` argument and `datagrid()` function in a `comparisons()` call.
All unspecified variables are held at their mean or mode. This includes grouping variables in mixed-effects models, so analysts who fit such models may want to specify the groups of interest using the `variables` argument, or supply model-specific arguments to compute population-level estimates. See details below.
See the "Plots" vignette and website for tutorials and information on how to customize plots:
- https://marginaleffects.com/articles/plot.html
- https://marginaleffects.com
## Parameters {.unnumbered}
`model`: (model object) Object fitted using the `statsmodels` formula API.
`variables`: (str, list, dictionary) Name of the variable whose contrast we want to plot on the y-axis. Refer to the `comparisons()` documentation.
`newdata`: (dataframe) When newdata is `None`, the grid is determined by the condition argument. When newdata is not `None`, the argument behaves in the same way as in the comparisons() function.
`condition`: (str, list, dictionary) Conditional comparisonss.
- Position's representation:
1. x-axis.
2. color.
3. facet (wrap if no fourth variable, otherwise cols of grid).
4. facet (rows of grid).
- Argument types:
- list : Names of the predictors to display
- Numeric variables in position 1 is summarized by 100 numbers
- Numeric variables in positions 2, 3 and 4 are summarized by Tukey’s five numbers
- dictionary : Keys correspond to predictors. Values can be one of the two below depending on predictor's type:
- Series or list of the same type as the original variable.
- Numeric variables:
- String: "minmax", "threenum", "fivenum".
- string : Same as list of length 1.
`by`: (bool, str, list) Marginal comparisonss.
Names of the categorical predictors to marginalize across. Max length of list is 4, with position meanings:
1. x-axis.
2. color.
3. facet (wrap if no fourth variable, otherwise columns of grid).
4. facet (rows of grid)
`wts`: (str, optional) Column name of weights to use for marginalization. Must be a column in `newdata`.
`vcov`: (bool, np.ndarray, default=True) Type of uncertainty estimates to report (e.g. for robust standard errors). Acceptable values are:
- `True`: Use the model's default covariance matrix.
- `False`: Do not compute standard errors.
- String: Literal indicating the kind of uncertainty estimates to return:
- Heteroskedasticity-consistent: `"HC0"`, `"HC1"`, `"HC2"`, `"HC3"`.
- np.ndarray: A custom square covariance matrix.
`transform`: (function) Function specifying a transformation applied to unit-level estimates and confidence intervals just before the function returns results. Functions must accept a full column (series) of a Polars data frame and return a corresponding series of the same length. Ex:
- `transform = numpy.exp`
- `transform = lambda x: x.exp()`
- `transform = lambda x: x.map_elements()`
`draw`: True returns a matplotlib plot. False returns a dataframe of the underlying data.
`gray`: True returns a gray scale adapted plot. False returns a plot in color. For the second position of the list in the `condition` or `by` argument, the list can have at most 5 elements.
---
# `plot_slopes()` {.unnumbered}
Plot slopes on the y-axis against values of one or more predictors (x-axis, colors/shapes, and facets).
The `by` argument is used to plot marginal slopes, that is, slopes made on the original data, but averaged
by subgroups. This is analogous to using the `by` argument in the `slopes()` function.
The `condition` argument is used to plot conditional slopes, that is, slopes made on a user-specified grid.
This is analogous to using the `newdata` argument and `datagrid()` function in a `slopes()` call.
All unspecified variables are held at their mean or mode. This includes grouping variables in mixed-effects models, so analysts who fit such models may want to specify the groups of interest using the `variables` argument, or supply model-specific arguments to compute population-level estimates. See details below.
See the "Plots" vignette and website for tutorials and information on how to customize plots:
- https://marginaleffects.com/articles/plot.html
- https://marginaleffects.com
## Parameters {.unnumbered}
`model`: (model object) Object fitted using the `statsmodels` formula API.
`variables`: (str, list, dictionary) Name of the variable whose marginal effect (slope) we want to plot on the y-axis. Refer to the `comparisons()` documentation.
`condition`: (str, list, dictionary) Conditional slopess.
- Position's representation:
1. x-axis.
2. color.
3. facet (wrap if no fourth variable, otherwise cols of grid).
4. facet (rows of grid).
- Argument types:
- list : Names of the predictors to display
- Numeric variables in position 1 is summarized by 100 numbers
- Numeric variables in positions 2, 3 and 4 are summarized by Tukey’s five numbers
- dictionary : Keys correspond to predictors. Values can be one of the two below depending on predictor's type:
- Series or list of the same type as the original variable.
- Numeric variables:
- String: "minmax", "threenum", "fivenum".
- string : Same as list of length 1.
`by`: (bool, str, list) Marginal slopess.
Names of the categorical predictors to marginalize across. Max length of list is 4, with position meanings:
1. x-axis.
2. color.
3. facet (wrap if no fourth variable, otherwise columns of grid).
4. facet (rows of grid)
`draw`: True returns a matplotlib plot. False returns a dataframe of the underlying data.
`newdata`: (dataframe) When newdata is `None`, the grid is determined by the condition argument. When newdata is not `None`, the argument behaves in the same way as in the slopes() function.
`wts`: (str, optional) Column name of weights to use for marginalization. Must be a column in `newdata`.
`vcov`: (bool, np.ndarray, default=True) Type of uncertainty estimates to report (e.g. for robust standard errors). Acceptable values are:
- `True`: Use the model's default covariance matrix.
- `False`: Do not compute standard errors.
- String: Literal indicating the kind of uncertainty estimates to return:
- Heteroskedasticity-consistent: `"HC0"`, `"HC1"`, `"HC2"`, `"HC3"`.
- np.ndarray: A custom square covariance matrix.
`gray`: True returns a gray scale adapted plot. False returns a plot in color. For the second position of the list in the `condition` or `by` argument, the list can have at most 5 elements.
---
# `fit_statsmodels()` {.unnumbered}
Fit a statsmodels model with output that is compatible with pymarginaleffects.
This function streamlines the process of fitting statsmodels models by:
1. Parsing the formula
2. Handling missing values
3. Creating model matrices
4. Fitting the model with specified options
## Parameters {.unnumbered}
`formula`: (str)
Model formula
* Example: "outcome ~ distance + incentive"
`data`: (pandas.DataFrame or polars.DataFrame) Dataframe with the response variable and predictors.
**Important:** All categorical variables must be explicitly converted to `Categorical` or `Enum` dtype before fitting. String columns are not accepted in model formulas.
For Polars DataFrames:
```python
import polars as pl
# Option 1: Cast to Categorical (simplest) {.unnumbered}
df = df.with_columns(pl.col("region").cast(pl.Categorical))
# Option 2: Cast to Enum with explicit category order (recommended for control) {.unnumbered}
categories = ["<18", "18 to 35", ">35"]
df = df.with_columns(pl.col("age_group").cast(pl.Enum(categories)))
```
For pandas DataFrames:
```python
df["region"] = df["region"].astype("category")
```
`engine`: (callable) statsmodels model class (e.g., OLS, Logit)
`kwargs_engine`: (dict, default={}) Additional arguments passed to the model initialization.
* Example: `{'weights': weights_array}`
`kwargs_fit`: (dict, default={}) Additional arguments passed to the model's fit method.
* Example: `{'cov_type': 'HC3'}`
## Returns {.unnumbered}
(ModelStatsmodels)
A fitted model wrapped in the ModelStatsmodels class for compatibility
with marginaleffects.
## Examples {.unnumbered}
```python
from marginaleffects import fit_statsmodels, get_dataset, predictions, slopes, comparisons
import statsmodels.api as sm
data = get_dataset("thornton")
# Model with robust standard errors {.unnumbered}
model_robust = fit_statsmodels(
formula="outcome ~ distance + incentive",
data=data,
engine=sm.OLS,
kwargs_fit={"cov_type": "HC3"}
)
predictions(model_robust)
slopes(model_robust)
comparisons(model_robust)
```
## Notes {.unnumbered}
The fitted model includes additional attributes:
- `data`: The processed data after listwise deletion
- `formula`: The original formula string
- `formula_engine`: Set to "statsmodels"
- `model`: The fitted statsmodels model object
---
# `fit_sklearn()` {.unnumbered}
Fit a sklearn model with output that is compatible with pymarginaleffects.
This function streamlines the process of fitting sklearn models by:
1. Parsing the formula
2. Handling missing values
3. Creating model matrices
4. Fitting the model with specified options
## Parameters {.unnumbered}
`formula`: (str)
Model formula
* Example: "outcome ~ distance + incentive"
`data`: (pandas.DataFrame or polars.DataFrame) Dataframe with the response variable and predictors.
**Important:** All categorical variables must be explicitly converted to `Categorical` or `Enum` dtype before fitting. String columns are not accepted in model formulas.
For Polars DataFrames:
```python
import polars as pl
# Option 1: Cast to Categorical (simplest) {.unnumbered}
df = df.with_columns(pl.col("region").cast(pl.Categorical))
# Option 2: Cast to Enum with explicit category order (recommended for control) {.unnumbered}
categories = ["<18", "18 to 35", ">35"]
df = df.with_columns(pl.col("age_group").cast(pl.Enum(categories)))
```
For pandas DataFrames:
```python
df["region"] = df["region"].astype("category")
```
`engine`: (callable) sklearn model class (e.g., LinearRegression, LogisticRegression)
`kwargs_engine`: (dict, default={}) Additional arguments passed to the model initialization.
* Example: `{'weights': weights_array}`
## Returns {.unnumbered}
(ModelSklearn)
A fitted model wrapped in the ModelSklearn class for compatibility
with marginaleffects.
## Examples {.unnumbered}
```{python}
from marginaleffects import *
from statsmodels.formula.api import ols
import polars as pl
import polars.selectors as cs
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder, FunctionTransformer
from sklearn.linear_model import LinearRegression
from sklearn.compose import make_column_transformer
from xgboost import XGBRegressor
# Linear regression: Scikit-learn {.unnumbered}
military = get_dataset("military")
# Convert categorical variables to proper dtypes {.unnumbered}
military = military.with_columns(
pl.col("branch").cast(pl.Categorical)
)
mod_sk = fit_sklearn(
"rank ~ officer + hisp + branch",
data=military,
engine=LinearRegression(),
)
avg_predictions(mod_sk, by="branch")
# Linear regression: Statsmodels {.unnumbered}
mod_sm = ols("rank ~ officer + hisp + branch", data=military.to_pandas()).fit()
avg_predictions(mod_sm, by="branch")
# XGBoost: Scikit-learn {.unnumbered}
airbnb = get_dataset("airbnb")
# Convert categorical variables to proper dtypes {.unnumbered}
catvar = airbnb.select(~cs.numeric()).columns
airbnb = airbnb.with_columns(
[pl.col(c).cast(pl.Categorical) for c in catvar]
)
train, test = train_test_split(airbnb)
def selector(data):
y = data.select(cs.by_name("price", require_all=False))
X = data.select(~cs.by_name("price", require_all=False))
return y, X
preprocessor = make_column_transformer(
(OneHotEncoder(), catvar),
remainder=FunctionTransformer(lambda x: x.to_numpy()),
)
pipeline = make_pipeline(preprocessor, XGBRegressor())
mod = fit_sklearn(selector, data=train, engine=pipeline)
avg_predictions(mod, newdata=test, by="unit_type")
avg_comparisons(mod, variables={"bedrooms": 2}, newdata=test)
```
## Notes {.unnumbered}
The fitted model includes additional attributes:
- `data`: The processed data after listwise deletion
- `formula`: The original formula string
- `formula_engine`: Set to "sklearn"
- `model`: The fitted sklearn model object
---
# `fit_linearmodels()` {.unnumbered}
Fit a linearmodels model with output that is compatible with pymarginaleffects.
This function streamlines the process of fitting linearmodels panel models by:
1. Parsing panel effects from the formula
2. Handling missing values
3. Creating model matrices
4. Fitting the model with specified options
## Parameters {.unnumbered}
`formula`: (str) Model formula with optional panel effects terms.
- Supported effects are:
- EntityEffects: Entity-specific fixed effects
- TimeEffects: Time-specific fixed effects
- FixedEffects: Alias for EntityEffects
- Example: `"y ~ x1 + x2 + EntityEffects"`
`data` : (pandas.DataFrame or polars.DataFrame) Panel data with MultiIndex (entity, time) or regular DataFrame with entity and time columns.
**Important:** All categorical variables must be explicitly converted to `Categorical` or `Enum` dtype before fitting. String columns are not accepted in model formulas.
For Polars DataFrames:
```python
import polars as pl
# Option 1: Cast to Categorical (simplest) {.unnumbered}
df = df.with_columns(pl.col("region").cast(pl.Categorical))
# Option 2: Cast to Enum with explicit category order (recommended for control) {.unnumbered}
categories = ["<18", "18 to 35", ">35"]
df = df.with_columns(pl.col("age_group").cast(pl.Enum(categories)))
```
For pandas DataFrames:
```python
df["region"] = df["region"].astype("category")
```
`engine`: (callable) linearmodels model class (e.g., PanelOLS, BetweenOLS, FirstDifferenceOLS)
`kwargs_engine`: (dict, default={}) Additional arguments passed to the model initialization.
* Example: `{'weights': weights_array}`
`kwargs_fit`: (dict, default={}) Additional arguments passed to the model's fit method.
* Example: `{'cov_type': 'robust'}`
## Returns {.unnumbered}
(ModelLinearmodels)
A fitted model wrapped in the ModelLinearmodels class for compatibility
with marginaleffects.
## Examples {.unnumbered}
```python
from linearmodels.panel import PanelOLS
from linearmodels.panel import generate_panel_data
from marginaleffects import *
data = generate_panel_data()
model_robust = fit_linearmodels(
formula="y ~ x1 + EntityEffects",
data=data.data,
engine=PanelOLS,
kwargs_fit={'cov_type': 'robust'}
)
predictions(model_robust)
```
## Notes {.unnumbered}
The fitted model includes additional attributes:
- `data`: The processed data after listwise deletion
- `formula`: The original formula string
- `formula_engine`: Set to "linearmodels"
- `model`: The fitted linearmodels model object
---
# `get_dataset()` {.unnumbered}
Download and read a dataset as a Polars DataFrame from the `marginaleffects` or from the list at https://vincentarelbundock.github.io/Rdatasets/.
Returns documentation link if `docs` is True.
## Parameters {.unnumbered}
`dataset`: (str) String. Name of the dataset to download.
- marginaleffects archive: affairs, airbnb, ces_demographics, ces_survey, immigration, lottery, military, thornton, factorial_01, interaction_01, interaction_02, interaction_03, interaction_04, polynomial_01, polynomial_02
- Rdatasets archive: The name of a dataset listed on the Rdatasets index. See the website or the search argument.
`package`: (str, optional) The package to download the dataset from.
`docs`: (bool, optional) If True, return the documentation URL instead of the dataset. Default is False.
`search`: (str, optional) The string is a regular expression. Download the dataset index from Rdatasets; search the "Package", "Item", and "Title" columns; and return the matching rows.
## Returns {.unnumbered}
(Union[str, pl.DataFrame])
* A string representing the documentation URL if `docs` is True, or
a Polars DataFrame containing the dataset if `docs` is False.
## Raises {.unnumbered}
ValueError
* If the dataset is not among the specified choices.
## Examples {.unnumbered}
```py
get_dataset()
get_dataset("Titanic", package="Stat2Data")
get_dataset(search = "(?i)titanic)
```