XAI-lecture 5

• permutation feature importance :

  global, model-agnostic
• Permutation feature imporance (+)

  • Intuitive
  • Highly compressed, global insight
  • Error ratio comparable across problems
  • Feature interactions accounted for
  • No retraining
• permutation feature importance :

  • Correlated features → biased by unrealistic samples
  • Need access to true labels
  • Randomness → results may vary
  • Adding corr. features decreases

  associated feature importance
• Partial depence plot (PDP)


  global, model-agnostic
• Partical dependence plot (+)

  • Intuitive and easy to interpret
  • Causal interpretatio
• Partial depence plot (-)

  • Assumes feature independence
  • Only for 1 to 2 features
  • Overinterpretation of regions with almost no data (add distribution!)
  • Heterogeneous effects may be hidden
• Individual conditional expectation (ICE)


  local, model-agnostic
• individual conditional expectation (+)

  • Even more intuitive than PDP
  • Heterogeneous relationships visible
• individual condition expectation (-)

  • Correlated features →unlikely or invalid data on the plot
  • Visual clutter
  • Hard to see average without added PDP
  • Can only display one feature
• Accumulated Local Effects (ALE) Plot

  global (but only local interpretation within intervals), model-agnostic
• Accumulated Local Effects (ALE) Plot (+)

  • Unbiased, also for correlated features
  • Faster computation
  • Conditional on a value
  • Centered at zero
  • Prediction function can be decompose
• Accumulated Local Effects (ALE) Plot (-)

  • Interpretation of the effect across intervals is not permissible
  • Need to balance interval size
  • Don’t work with ICEs
  • Harder to implemen
• Neural Additive Models (NAMS)


  globally + intrinsically interpreteble, model-specific
• Neural additive models (NAM) (+)

  • Complete description of the model
  • Allows the application of any Neural Network architecture
  • Allows for multitask prediction
  • Differentiability
• Neural additive models (NAMS) (-)

  • No feature interactionsCould add x1 * x2 to learn simple
  • feature interactions, but this gradually makes interpretation more difficult
• Global surrogate models 


  global, model-agnostic
• Global surrogate models (+)

  • flexible
  • intuitive
• Global surrogate models (-)

  • closeness to actual model sometimes hard to interpret
• Local Surrogate Models


  local, model-agnostic
• local surrogate models (+)

  • Work for tabular data, images, and text
  • Can use more easily interpretable features
  • Fast and easy to use
• Local surrogate models (-)

  • difficult to choose neighborhood correctly
  • instable and prone to adversarial attacks
• Shapley values

  local, model-agnostic
• shapley values (+)

  • Attributions fairly distributed
  • Theoretically grounded
  • Allows contrastive explanations
• shapely values (-)

  • Computationally expensive
  • Must use all features
  • No prediction model
• Neirest Neighbor examples


  local, model-agnostic
• nearest neighbor examples (+)

  • Data-agnostic
  • Highly intuitive
  • Seem to do well in user experiments
• Nearest neighbor examples (-)

  • Difference between measuring in latent andinput space might cause confusion
  • Needs access to ground truth to be effective
  • Unclear how many neighbors to show
• Saliency Maps 


  local, model specific
• saliency maps additinal techniques

  guided grad-cam, smoothgrad
• Saliency maps (+)

  • visual and intuitive
  • fast to compute
• saliency maps (-)

  • Difficult to judge correctness
  • Prone to adversarial attacks
  • Guided methods can fail insensitivity tests
• Network dissection

  local/global dependant on usage, model-specific
• network dissection (+)

  • Links units to concepts
  • Non-technical output
  • Can be combined with feature attribution
• network dissection (-)

  • Many units to look at
  • Units might not be interpretable
  • Requires pixel-level labels or good segmentation mode
• influental instances

  global

  deletion diagnostics : model-agnostic

  influence functions : model-specific
• influential instances (+):

  • Great for debugging
  • Show robustness of model
  • Difference can be measured in
  • training loss
  • model parameters
  • model predictions
  • other XAI measures
• influential instances (-)

  • Computationally expensive due to retraining
  • Influence functions only approximate
  • No clear cut-off between influential and non-influential
• Prototypes and criticsms

  global, model-agnostic
• prototypes and criticisms (+)

  • describe dataset
  • free to choose number of prototypes
• prototypes and cristicsms (-)

  • distinction based on cutoff value
  • free to choose number of prototypes