08.interpretations.md

Toward Explicit Modeling of Implicit Adaptivity: Local Models, Surrogates and Post Hoc Approximations

Motivation

Building on the prior discussion of implicit adaptivity, this section examines methods that expose, approximate, or control those adaptive mechanisms.
Implicit adaptivity allows powerful models, including foundation models, to adjust behavior without explicitly representing a mapping from context to parameters [@doi:10.48550/arXiv.2108.07258]. This flexibility obscures the underlying mechanisms of adaptation, hindering modular reuse and systematic auditing. Making adaptivity explicit improves alignment with downstream goals, enables modular composition, and supports debugging and error attribution. It also fits the call for a more rigorous science of interpretability with defined objectives and evaluation criteria [@doi:10.48550/arXiv.1702.08608; @doi:10.48550/arXiv.2402.02870].
This chapter reviews practical approaches for surfacing structure, the assumptions they rely on, and how to evaluate their faithfulness and utility.

From Implicit to Explicit Adaptivity
Implicit adaptivity is hidden, flexible, and hard to audit, while explicit adaptivity surfaces modular structure that is structured, auditable, and controllable. The transition highlights three key trade-offs developed in this section: Fidelity vs. Interpretability, Local vs. Global Scope, and Approximation vs. Control.

{#fig:implicit-to-explicit width="80%"}

Approaches

Efforts to make implicit adaptation explicit span complementary strategies that differ in assumptions, granularity, and computational cost. We group them into six families:

1. surrogate modeling for local approximation,
2. prototype- and neighbor-based reasoning,
3. diagnostics for amortized inference,
4. disentanglement and bottleneck methods,
5. parameter extraction and probing, and
6. emerging approaches that leverage large language models as post-hoc explainers.

Surrogate Modeling

This line of work approximates a black-box $h(x,c)$ with an interpretable model in a small neighborhood, so that local behavior and a local view of $f(c)$ can be inspected. A formal template is

$$ \hat{g}_{x_0,c_0} = \arg\min_{g \in \mathcal{G}} , \mathbb{E}_{(x,c) \sim \mathcal{N}_{x_0,c_0}} \left[ \ell\big(h(x,c), g(x,c)\big) \right] + \Omega(g), $$

where $\mathcal N_{x_0,c_0}$ defines a locality (e.g., kernel weights), $\ell$ measures fidelity, and $\Omega$ controls complexity. Where $\mathcal{G}$ denotes a restricted hypothesis class, often composed of linear or other low-complexity functions chosen to enhance interpretability. A convenient local goodness-of-fit is

$$ R^2_{\text{local}} = 1 - \frac{\sum_i w_i,\big(h_i - g_i\big)^2}{\sum_i w_i,\big(h_i - \bar h\big)^2}, \qquad w_i \propto \kappa!\big((x_i,c_i),(x_0,c_0)\big). $$

LIME perturbs inputs and fits a locality-weighted linear surrogate [@doi:10.48550/arXiv.1602.04938]; SHAP / DeepSHAP provide additive attributions based on Shapley values [@doi:10.48550/arXiv.1705.07874]. Integrated Gradients and DeepLIFT link attribution to path-integrated sensitivity or reference-based contributions [@doi:10.48550/arXiv.1703.01365; @doi:10.48550/arXiv.1704.02685]. These methods are most reliable when the model is near-linear in the chosen neighborhood and perturbations remain near the data manifold; consequently, a rigorous analysis involves stating the neighborhood definition, reporting the surrogate’s goodness-of-fit, and assessing stability across seeds and baselines.

Prototype and Nearest-Neighbor Methods

Here, a decision is grounded by reference to similar cases in representation space, which supports case-based explanations and modular updates. ProtoPNet learns a library of visual prototypes to implement “this looks like that” reasoning [@doi:10.48550/arXiv.1806.10574]. Deep $k$-nearest neighbors audits predictions by querying neighbors in activation space and can flag distribution shift [@doi:10.48550/arXiv.1803.04765]. Influence functions link a prediction to influential training points for data-centric debugging [@doi:10.48550/arXiv.1703.04730]. This line of work connects naturally to exemplar models and contextual bandits, where decisions are justified via comparisons to context-matched exemplars. Reports include prototype coverage and diversity, neighbor quality checks, and the effect of editing prototypes or influential examples. These prototype-based approaches make local adaptation explicit by grounding predictions in reference cases, bridging the gap between black-box models and case-based reasoning frameworks.

Amortization Diagnostics

For amortized inference systems (e.g., VAEs), the encoder $q_{\phi}(\theta\mid x)$ can be treated as an implicit $f(c)$. Diagnostics measure amortization gaps and identify suboptimal inference or collapse [@doi:10.48550/arXiv.1801.03558]. Useful outputs include calibration under shift and posterior predictive checks, together with ablations that vary encoder capacity or add limited iterative refinement. This clarifies when the learned mapping is faithful versus when it underfits the target posterior. Such diagnostics mirror classical checks for approximate Bayesian inference, where amortization gaps quantify the discrepancy between learned and exact posteriors.

Disentangled and Bottlenecked Representations

While amortization diagnostics target model faithfulness, disentanglement aims to expose interpretable subspaces aligned with distinct contextual factors. The aim is to expose factors that align with distinct contextual causes, making changes traceable and controllable. $\beta$-VAE encourages more factorized latents [@higgins2017betavae], while the Deep Variational Information Bottleneck promotes predictive minimality that can suppress spurious context [@doi:10.48550/arXiv.1612.00410]. Concept-based methods such as TCAV and ACE map latent directions to human concepts and test sensitivity at the concept level [@doi:10.48550/arXiv.1711.11279; @doi:10.48550/arXiv.1902.03129]. Fully unsupervised disentanglement is often ill-posed without inductive bias or weak supervision [@doi:10.48550/arXiv.1811.12359]. Quantitative evaluation of disentanglement can follow established metrics that assess factor independence, completeness, and informativeness [@eastwood2018a]. Reports should include concept validity tests, factor stability across runs, and simple interventions that demonstrate controllability.

Parameter Extraction and Probing

This family locates where adaptation is encoded and exposes handles for inspection or edits. Linear probes test what is linearly decodable from intermediate layers [@doi:10.48550/arXiv.1610.01644]; edge probing examines specific linguistic structure in contextualized representations [@doi:10.48550/arXiv.1905.06316]. Model editing methods such as ROME can modify stored factual associations directly in weights [@doi:10.48550/arXiv.2202.05262], while “knowledge neurons” seek units linked to particular facts [@doi:10.48550/arXiv.2104.08696]. Evaluation involves quantifying pre- and post-edit behavior, assessing locality and persistence, and documenting side effects on unrelated capabilities. Collectively, these methods transform hidden internal adaptations into analyzable modular components.

LLMs as Post-hoc Explainers

Recent work uses in-context prompting to elicit rationales, counterfactuals, or error hypotheses from large language models for a target system [@doi:10.48550/arXiv.2310.05797]. These explanations can be useful but must be validated for faithfulness, for example by checking agreement with surrogate attributions, reproducing input–output behavior, and testing stability to prompt variations. Explanations should be treated as statistical estimators with stated objectives and evaluation criteria [@doi:10.48550/arXiv.2402.02870].

These methodological families differ in their assumptions and computational granularity, yet they all aim to render adaptation transparent and controllable. The following sections summarize their key trade-offs and conceptual challenges.

Trade-offs

Fidelity vs. Interpretability

High-fidelity surrogates capture the target model’s behavior more accurately, yet they often grow in complexity and lose readability. A crisp statement of the design goal is

$$ \min_{g\in\mathcal G}\ \underbrace{\phi_{\text{fid}}(g;U)}_{\text{faithfulness on use set }U}

• \lambda\underbrace{\psi_{\text{simplicity}}(g)}_{\text{sparsity / size / semantic load}}, $$

where $\phi_{\text{fid}}$ can be local $R^2$, AUC, or rank correlation with $h$, and $\psi_{\text{simplicity}}$ can be sparsity, tree depth, rule count, or active concept count. If a simple surrogate underfits, consider structured regularization (e.g., monotonic constraints, grouped sparsity, concept bottlenecks). If a complex surrogate is needed, accompany it with readable summaries (partial dependence snapshots, distilled rule sets, compact concept reports).

Local vs. Global Scope

Local surrogates aim for $g_{x_0,c_0}\approx h$ only on $\mathcal N_{x_0,c_0}$, whereas a global surrogate seeks $g_{\text{global}}\approx h$ across the domain, potentially smoothing away distinct regimes. Hybrid schemes combine both:

$$ g(x,c)=\sum_{k=1}^{K} w_k(x,c), g_k(x,c), \qquad \sum_k w_k(x,c)=1,\quad w_k\ge 0, $$

with local experts $g_k$ and soft assignment $w_k$. Report the neighborhood definition, coverage (fraction of test cases with acceptable local fit), and disagreements between local and global views; flag regions where the global surrogate is unreliable.

Approximation vs. Control

Coarse modularization makes control and auditing simpler because edits act on a small number of levers, yet residual error can be large. Fine-grained extraction, such as neuron- or weight-level edits, can achieve precise behavioral changes but may introduce unintended side effects. Define the intended edit surface in advance (concepts, features, prototypes, submodules, parameters). For coarse modules, measure the residual gap to the base model and verify that edits improve target behavior without harming unaffected cases. For fine-grained edits, quantify locality and collateral effects using a held-out audit suite with counterfactuals, canary tasks, and out-of-distribution probes. Maintain versioned edits, enable rollback, and document the scope of validity.

These trade-offs are not merely design choices but determine the operational boundaries within which explicit representations can remain faithful to the original adaptive system.

Open Research Directions

Reusable Modules

The challenge of isolating reusable routines parallels the quest for parameter-efficient fine-tuning in large models, where adaptation must remain modular yet composable. A central question is whether we can isolate portable skills or routines from large models and reuse them across tasks without degrading overall capability [@doi:10.48550/arXiv.2108.07258]. Concretely, a reusable module should satisfy portability, isolation, composability, and stability. Promising directions include concept bottlenecks that expose human-aligned interfaces, prototype libraries as swappable reference sets, sparse adapters that confine changes to limited parameter subsets, and routing mechanisms that select modules based on context. Evaluation should track transfer performance, sample efficiency, interference on held-out capabilities, and robustness under domain shift.

Performance Gains

When does making structure explicit improve robustness or efficiency compared to purely implicit adaptation? Benefits are most likely when domain priors are reliable, data are scarce, or safety constraints limit free-form behavior. Explicit structure is promising when context topology is known (spatial or graph), when spurious correlations should be suppressed, and when explanations must be auditable. To assess this, fix capacity and training budget and vary only the explicit structure (prototypes, disentanglement, bottlenecks). Stress tests should cover diverse distributional challenges, including covariate shift, concept shift, long-tail classes, and adversarially correlated features. Account for costs such as concept annotation, extra hyperparameters, and potential in-domain accuracy loss.

Abstraction Level

Another open issue is the appropriate level at which to represent structure: parameters (weights, neurons), functions (local surrogates, concept scorers, routing policies), or latent causes (disentangled or causal factors). Benchmarking under fixed capacity and identical data regimes is essential to isolate the contribution of explicit structure from mere model scaling effects. Choose based on the use case. For safety patches, lower-level handles allow precise edits but require guardrails and monitoring. For scientific or policy communication, function- or concept-level interfaces are often more stable and auditable. Optimize three objectives in tension: faithfulness to the underlying model, usability for the target audience, and stability under shift. Tooling should support movement between levels (e.g., distilling weight-level edits into concept summaries or lifting local surrogates into compact global reports). Selecting the proper level of abstraction thus defines not only interpretability but also the feasible scope of control.

Evaluation and Reporting Standards for Classical Post-hoc Methods

LIME, SHAP, and gradient-based methods such as Integrated Gradients and DeepLIFT remain common tools for context-adaptive interpretation. Their usefulness depends on careful design and transparent reporting. Explanations should be treated as statistical estimators with stated objectives and evaluation criteria [@doi:10.48550/arXiv.1702.08608; @doi:10.48550/arXiv.2402.02870]. Carmichael & Scheirer (2021) further propose a principled evaluation framework for feature-additive explainers, enabling measurement of misattribution even under known ground-truth additive models [@doi:10.48550/arXiv.2106.08376].

Scope and locality

Local surrogate methods require a clear definition of the neighborhood in which the explanation is valid. The sampling scheme, kernel width, and surrogate capacity determine which aspects of the black box can be recovered. When context variables are present, the explanation should be conditioned on the relevant context and the valid region should be described.

Attribution methods in practice

Attribution based on gradients is sensitive to baseline selection, path construction, input scaling, and preprocessing. Baselines should have clear domain meaning, and sensitivity analyses should show how conclusions change under alternative baselines. For perturbation-based surrogates, report the perturbation distribution and any constraints that keep samples on the data manifold.

Faithfulness and robustness

Faithfulness and robustness should be checked rather than assumed. Useful checks include deletion and insertion curves, counterfactual tests, randomization tests, stability under small input and seed perturbations, and for local surrogates a local goodness-of-fit such as a neighborhood $R^2$. The evaluation metric should match the stated objective of the explanation [@doi:10.48550/arXiv.1702.08608; @doi:10.48550/arXiv.2402.02870]. Turbé et al. (2023) demonstrate evaluation of interpretability methods on time-series models using metrics such as $\widetilde{\mathrm{AUC}}{S}$ and $\widetilde{F{1,S}}$ to compare alignment with model internals [@doi:10.1038/s42256-023-00620-w].

Minimal reporting checklist

Item	Description
Data slice and context definition	Specify the subset of data and contextual variables used for generating explanations, and describe the locality or neighborhood definition.
Surrogate specification and regularization details	Report the family of surrogate models, chosen regularization strategy, and kernel or sampling parameters.
Faithfulness and robustness metrics	Include local $R^2$, deletion/insertion area, counterfactual validity, and robustness under perturbations.
Sensitivity and uncertainty analysis	Assess variation across baselines, random seeds, and small input perturbations, providing uncertainty estimates.
Computational constraints	Document runtime, hardware limitations, and approximation budgets that affect explanation quality.
Observed limitations and failure modes	Summarize known weaknesses, unstable regions, or interpretability failures identified during validation.

Table 2. Minimal Reporting Checklist for Post-hoc Explanations

From post hoc analysis to design

Insights from post-hoc analysis can inform proactive model design for control, auditing, and policy communication. In such cases, interpretability methods should not remain external diagnostics but serve as guides for architectures with built-in transparency. For example, Concept Bottleneck Models integrate interpretable concepts into the forward pass [@doi:10.48550/arXiv.2007.04612]. Similarly, Poursabzi-Sangdeh et al. (2021) conduct empirical user studies to highlight how interpretability design choices affect human use and model trust [@doi:10.48550/arXiv.1802.07810]. These contributions extend the vision of Doshi-Velez & Kim (2017) toward a unified science of interpretable modeling, where explanation and model training are co-designed [@doi:10.48550/arXiv.1702.08608]. Taken together, these lines of work bridge black-box adaptation and structured inference and set the stage for designs where context-to-parameter mappings are specified, trained, and evaluated end to end.

Implications for classical models

These tools can also clarify how traditional models, for example, logistic regression with interaction terms or generalized additive models to admit a local adaptation view: a simple global form paired with context-sensitive weights or features. Reading such models through the lens of local surrogates and concept interfaces helps align classical estimation with modern, context-adaptive practice. Reinterpreting these classical estimators through the lens of explicit adaptivity situates them as early instances of structured context modeling, underscoring continuity between statistical modeling and modern machine learning.

Taken together, these strategies illustrate a gradual unification of interpretability, modularization, and adaptive modeling, paving the way toward a principled science of explicit context-aware inference.