Some Reflections on Drawing Causal Inference using Textual Data: Parallels Between Human Subjects and Organized Texts | Summary

This article explains the key points of Some Reflections on Drawing Causal Inference using Textual Data: Parallels Between Human Subjects and Organized Texts.

• 2022-02-02 (arXiv), 1st Conference on Causal Learning and Reasoning
• Zhang, Bo, Zhang, Jiayao.
• Department of Statistics and Data Science, University of Pennsylvania, University of Pennsylvania
• Paper

Read this article in Korean

Summary

• The paper examines how to frame causal questions when documents themselves are the units being studied. Through conceptual comparisons with studies of human subjects, it argues that defining interventions on perceived traits or their constituent parts can clarify ambiguous causal questions; it reports no experimental performance comparison.
• Using conference-paper acceptance and Reddit post popularity as examples, the authors emphasize that adjustment variables must be unaffected by the specified treatment. They distinguish directly observable text features from inferred features and warn that measurement error and unobserved confounding can undermine causal conclusions.

1. Introduction: Causal Inference with Textual Data

Text can convey causal knowledge, encode variables in studies of people, or itself constitute the study units whose attributes are under investigation. This essay focuses on the third role and uses the potential outcomes framework, which defines a causal effect by comparing outcomes under alternative treatments.

2. Study Units and Covariates

An estimand is the quantity a study aims to estimate; the authors recommend defining it and the study units before specifying an algorithm. In the conference-paper example from Veitch et al. (2020), subject matter may influence both the presence of a theorem and acceptance, while the abstract is proposed as a source of information about that subject.

Covariates are variables determined before treatment or otherwise unaffected by it; adjusting for variables affected by treatment can introduce posttreatment bias, a distortion caused by such adjustment. Whether text features qualify therefore depends on the treatment being defined.

3. Immutable Characteristics and Causal Variables

The authors question whether the number of theorems is a well-defined treatment: some papers cannot plausibly receive a theorem, and adding a convergence theorem differs from adding a minimax lower-bound theorem. These differing versions raise a concern about the stable unit treatment value assumption, which requires a sufficiently well-defined treatment.

Words describing a theorem, including in an abstract, may change with the theorem and thus may be unsuitable adjustment variables; the same concern applies to embeddings, numerical representations derived from those words. The authors suggest restricting attention to plausible interventions and unaffected metadata, while describing this as an imperfect solution.

4. Two Strategies Facilitating Framing Proper Causal Queries

The paper proposes two ways to clarify a causal query: intervene on a decision maker’s perception of a trait, and define a broader concept through its constituent traits. Both require specifying the treatment and determining which features remain eligible for adjustment.

4.1. Shifting from actual traits to perceptions of them

For Reddit posts, the authors propose manipulating a gender-indicating hashtag or icon to change viewers’ perceptions while leaving the writer’s gender identity unchanged, following the perceived-traits framework of Greiner and Rubin (2011). Under this formulation, unchanged post content can be treated as a pretreatment covariate, whereas viewers’ subsequent comments are posttreatment variables.

4.2. Shifting from one trait to a property/concept including many traits

The authors suggest studying perceived mathematical rigor as a bundle that can include theorems, graphs, paper flow, and references to theory papers. For perceived politeness, sentiment and formality may be components of the treatment, while topic, authorship, and brevity may be covariates; the distinction requires domain knowledge and may remain ambiguous.

5. A Dichotomy of Covariates

Given a defined causal query, explicit observed covariates include directly obtainable counts such as numbers of figures, while implicit observed covariates include inferred sentiment or text representations. The appropriate set depends on the query, and language-model-derived features may be measured inaccurately or summarize the text insufficiently.

Observational causal inference depends on assumptions about unmeasured confounding, meaning omitted factors that affect both treatment assignment and outcomes. The authors suggest exploring sensitivity analysis, which assesses how hidden confounding could affect conclusions, negative controls, which help detect bias, and instrumental variable methods, which use an external variable under identifying assumptions, because relevant semantic information may remain unrecovered.

6. Conclusion

The authors recommend identifying study units, specifying the treatment and its timing or components, and justifying each adjustment variable. They caution that a question without even a hypothetical experimental analogue may be ill-posed or unidentified, meaning its causal effect cannot be determined from the available information, and that these steps expose problems without guaranteeing a solution.

Brief Thoughts

The examples make the distinction between a textual feature and a feasible intervention concrete: adding a theorem can also change the abstract, while changing a gender label can leave the post itself fixed. The paper is a conceptual essay, so it does not establish empirically whether its proposed formulations reduce bias in a particular dataset.