The Compression Tax: An Audit of How Post-Training Quantization Redistributes Safety and Fairness Failures in Open Language Models
Amit Sagtani (Alyf Inc) and Saloni Garg (Adobe). ACM FAccT 2027 submission #1159 — abstract registered October 27, 2026; full paper in preparation for the November 3, 2026 deadline.
When an open language model is quantized after training, do the resulting safety and fairness costs fall evenly, or do they concentrate in already disadvantaged groups and in specific refusal behaviors? Averages can hide the answer: an aggregate score can stay flat while harm moves onto the groups least able to absorb it. This audit measures the deltas per group, not just on average.
Three open model families at matched scale (Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3), each at BF16, INT8, and 4-bit NF4 through one quantization stack. Five evaluation suites, every endpoint disaggregated: BBQ fairness across all nine bias categories; HarmBench harmful-prompt compliance; XSTest over-refusal on safe prompts; AAVE/SAE dialect pairs; RealToxicityPrompts toxicity by demographic slice. Four pre-registered hypotheses (concentration, directional refusal, bit-width monotonicity, masking), paired bootstrap confidence intervals with Holm correction.
The paper proposes a compression delta disclosure for model cards: whenever a quantized variant is released, its per-group safety and fairness deltas against the full-precision model are reported alongside the average scores — so a compression that passes on average but concentrates harm is visible before deployment.