CLIP Image Encoder Fine-Tuning Hurts Cross-Domain AI Models, IJCAI Study Finds

Aggressively fine-tuning CLIP’s image and text encoders for task-specific deployment — one of the most common practices in production vision-language AI — actively degrades model performance on out-of-distribution data, according to a new paper being presented this week at IJCAI-ECAI 2026 in Bremen, Germany. The finding, from a team at Zhejiang University and Swansea University, directly contradicts the intuition that has guided a generation of practitioners adapting CLIP-based models to specialized domains: if your task is visual, tune the visual encoder harder.

The paper, A₃B₂: Adaptive Asymmetric Adapter for Alleviating Branch Bias in Vision-Language Image Classification with Few-Shot Learning, identifies what the authors call Branch Bias — a systematic asymmetry, previously unstudied in general image classification, in how CLIP’s image and text encoders respond to fine-tuning. The solution, an uncertainty-driven adapter method that automatically suppresses image-branch modifications when a model is likely encountering unfamiliar data, outperforms eleven competitive baselines across all three standard few-shot evaluation protocols.

The Zhejiang University paper demonstrates this intuition is systematically wrong for a specific and practically important condition: when the model will be deployed across different data distributions than those it was trained on — the normal condition for any production AI system serving a mixed real-world input stream.

Why Practitioners Reached for the Image Encoder First

CLIP, trained on 400 million image-text pairs and released by OpenAI in 2021, became the backbone of production vision-language systems by learning to align visual and textual representations in a shared embedding space through contrastive learning. When practitioners subsequently adapt CLIP to specialized domains with limited labeled data — a scenario that describes the majority of real-world fine-tuning use cases — the standard intuition has been intuitive to the point of feeling obvious: because the task involves images, the image encoder must be trained more. Add adapters to the vision branch, tune them on your target dataset, and the model should capture the domain’s visual details more precisely.

Three Findings That Overturn the Standard Practice

Lead researcher and corresponding author Professor Jingyuan Chen and co-authors Yiyun Zhou, Zhonghua Jiang, Wenkang Han, Kunxi Li, Chang Yao, and Mingjing Xu (Swansea University) ran controlled experiments attaching basic adapter modules to either the image or text encoder of three CLIP variants (ViT-B/32, ViT-B/16, and ViT-L/14) and measured performance across 11 benchmark datasets and three evaluation protocols.

The results produced three findings that the authors describe as overturning the dominant paradigm:

Text-side adapters deliver more reliable gains. Across most tasks, adapting the text encoder with a comparable parameter budget produced substantially larger performance improvements than adapting the image encoder. The paper’s theoretical explanation: text encodes high-level semantic concepts, and a small text-side adapter can quickly realign the model’s concept space to new categories. The image encoder, in contrast, already encodes robust visual representations built from 400 million diverse examples — representations that are harder to improve incrementally than to corrupt.

When training and test distributions match, tuning both branches is optimal. In base-to-novel generalization experiments — where the model is trained on seen categories and evaluated on unseen categories from the same dataset — simultaneously fine-tuning both image and text encoders achieved the best results. In this in-distribution setting, image-encoder fine-tuning does provide a genuine benefit by capturing fine-grained domain-specific visual features.

For out-of-distribution tasks, image-encoder fine-tuning is actively harmful. This is the finding with the most direct practical consequences. In cross-dataset transfer and domain generalization experiments — where test data comes from a different distribution than training data — aggressively fine-tuning the image encoder did not merely fail to help but measurably degraded performance. The pre-trained image encoder’s broad visual representations, built from 400 million diverse image-text pairs, were corrupted by forcing them to adapt to a narrow target domain. The model became, in the paper’s language, "near-sighted."

The data is explicit. In cross-dataset evaluation, MMA — a prior state-of-the-art method that simultaneously adapts both encoders — averaged 63.65 across ten transfer datasets. The A₃B₂ method averaged 66.39 on the same benchmarks. In domain generalization across ImageNet variants with severe distribution shifts (ImageNet-V2, ImageNet-Sketch, ImageNet-A, ImageNet-R), A₃B₂ averaged 60.81 while MMA averaged 59.21.

What Does the Softmax Maximum-Probability Heuristic Measure — and When Does It Fail?

How CLIP’s confidence is measured matters here. A₃B₂’s uncertainty-aware dampening relies on the maximum softmax probability as its confidence proxy: the highest probability value assigned to any class. When that value is high, the model is confident, the sample is probably in-distribution, and the image adapter applies normally. When it is low, the model is uncertain, the sample is probably out-of-distribution, and a regularization penalty forces the image adapter’s output toward zero — effectively reverting the model to CLIP’s robust pre-trained image representations for that input.

The research builds on Hendrycks and Gimpel’s foundational out-of-distribution detection work (2016), who established maximum softmax probability as a baseline out-of-distribution detector. An important limitation is worth naming: softmax max-probability is known in the uncertainty quantification literature to be overconfident on out-of-distribution inputs. A model can be wrong with high confidence. If a CLIP model has overfit to the wrong features of an out-of-distribution input and assigns it high softmax probability, the UAAD mechanism would not suppress the image adapter — precisely when suppression would help most. The paper demonstrates strong empirical results across standard benchmarks but does not test UAAD’s behavior in adversarial out-of-distribution regimes where softmax overconfidence is most severe.

The Inescapable Dilemma — and Why a Rigid Solution Fails

Once the Branch Bias finding is clear, the naive response is an obvious rule: lock the image encoder whenever you expect out-of-distribution data, and leave it trainable otherwise. This is exactly what the paper argues cannot work.

Real production inference streams do not arrive labeled as "in-distribution" or "out-of-distribution." A model serving e-commerce product search, satellite imagery analysis, or medical imaging will inevitably encounter a mixture of familiar and unfamiliar inputs with no prior flag distinguishing them. There is no practical mechanism to toggle the image adapter on and off based on incoming data — not in real-time production, and not even as an offline preprocessing step, because "distribution" is a property of the aggregate data stream, not of individual inputs.

This is the dilemma the paper’s core method resolves by moving the control decision from the engineer to the data itself.

How A₃B₂ Works: Uncertainty Drives the Brakes

The Adaptive Asymmetric Adapter (A₃B₂) replaces the engineer’s toggle with a continuous, differentiable soft constraint. Its Uncertainty-Aware Adapter Dampening (UAAD) mechanism works as follows:

After each forward pass, the model computes the softmax probability distribution over classes. The maximum probability value (κ) serves as a confidence score. A regularization loss (L_bias) is then applied that scales with (1 − κ): for high-confidence inputs, the image adapter’s output is applied at full strength; for low-confidence inputs, the penalty forces the image adapter’s contribution toward zero, reverting the model to CLIP’s pre-trained visual representations. The text adapter is never subject to this dampening. It remains fully active for every input, consistent with the paper’s first insight that text-side adaptation provides reliable gains regardless of distribution. The asymmetry is structural: one branch has a brake; the other does not.

The adapter’s internal architecture introduces a second level of asymmetry, inspired by Mixture-of-Experts principles. Each adapter layer uses a "one down, many up" structure: a single shared dimensionality-reduction matrix compresses features into a low-dimensional representation, which is then routed to multiple specialized "up expert matrices" by a dynamic router. The paper provides an information-theoretic justification using the Information Bottleneck principle: a single shared bottleneck forces the compressed representation to encode only task-relevant information, avoiding the credit-assignment conflicts that plague the reverse "many downs, one up" configuration. A Load Balancing Regularization term (L_bal) prevents expert collapse by penalizing the router for deviating from uniform expert assignment across a batch.

Benchmark Results Across Three Protocols

The team evaluated A₃B₂ against eleven competitive baselines — CoOp, CoCoOp, KgCoOp, MaPLe, CLIP-Adapter, RPO, TCP, MMA, MMRL, and MMRL++ — across 11 datasets in the 16-shot setting.

In base-to-novel generalization (in-distribution), A₃B₂ achieved the highest average Harmonic Mean score across all 11 datasets (80.49), surpassing MMRL++ (79.29), MMA (78.99), and MaPLe (78.27). Gains were most pronounced on fine-grained visual datasets: StanfordCars (+3.43 HM over MaPLe), EuroSAT (+0.93 HM over MMRL++), and FGVCAircraft (+5.31 HM over MaPLe).

In cross-dataset evaluation (out-of-distribution transfer), A₃B₂ matched or led on most of the 10 transfer datasets, averaging 66.39. MMA — which simultaneously adapts the image encoder on out-of-distribution data — performed markedly worse (63.65 average), exactly as predicted by the paper’s third insight.

In domain generalization (severe distribution shift), A₃B₂ averaged 60.81 across four ImageNet variants, outperforming all 11 baselines. Text-only optimization methods (CoOp, KgCoOp) substantially outperformed image-encoder optimization methods (MMA, MaPLe) here, further validating the Branch Bias hypothesis.

Ablation experiments confirmed that removing the L_bias dampening mechanism caused significant drops in novel-class performance. Applying the same dampening to the text encoder also hurt performance substantially — validating the asymmetric design: the brake belongs only on the image branch. The method also proved compatible with CNN-based CLIP backbones (ResNet-50), not just Vision Transformers, suggesting architectural generalizability.

What Practitioners Should Change — and What the Method Does Not Solve

The paper translates its findings into engineering guidance. For practitioners adapting CLIP-based vision-language models in production with limited labeled data:

Start with the text encoder, not the image encoder. Text-side adapters are cheaper, converge faster, and are more robust to distribution shift. This should be the default starting point.

In-distribution fine-tuning allows both branches. When there is high confidence that deployment data closely matches training data, adapting both encoders remains valid and achieves the best results.

For unknown or mixed distributions, image adapters are a liability. Standard image-encoder fine-tuning in cross-domain or open-world deployment introduces a performance ceiling and a robustness floor that A₃B₂ is designed to avoid.

The method functions as a safety net. Rather than aggressively fitting unfamiliar inputs, A₃B₂ gracefully degrades to CLIP’s pre-trained generalization when confidence is low — a natural defense against the "confidently wrong" failure mode in deployed models.

The paper’s limits are also explicit. In scenarios with millions of carefully annotated target-domain images, suppressing the image adapter may constrain peak in-distribution performance. The method is explicitly optimized for constrained-compute, few-shot transfer settings — the category of deployment that describes most real-world industrial and edge applications, but not every one.

The paper also fits within a growing body of research documenting CLIP fine-tuning’s out-of-distribution trade-off. CLIPood (Shu et al., 2022) showed that adapting CLIP to specific tasks costs out-of-distribution generalization. The VL2V paper (2023) found text embeddings preserve OOD robustness better than image embeddings in vision-language models. A₃B₂’s contribution is to name the asymmetry, quantify it systematically across multiple CLIP variants and 11 datasets, and provide a practical soft-constraint solution that requires no prior knowledge of the test distribution.

The research was funded by Zhejiang Province’s "Pioneer" and "Leading Goose" R&D Program (Grant No. 2025C02022) and the National Natural Science Foundation of China (No. 62307032). Code is publicly available at https://github.com/zyy-2001/A3B2, and the paper is at https://arxiv.org/abs/2605.13161.

---

Frequently Asked Questions

Does this finding mean image-encoder fine-tuning should never be done?

No — but it should be done with caution about deployment context. The paper’s second insight shows that when training and test data come from the same distribution, fine-tuning both encoders simultaneously achieves the best results. The warning applies specifically to out-of-distribution deployment: if a model trained on domain A will encounter inputs from domain B at inference time (and most production models will), image-encoder fine-tuning is likely to degrade performance on domain B. The A₃B₂ method handles both conditions simultaneously by letting uncertainty drive the image adapter’s contribution.

How does CLIP fine-tuning degrade a model that seemed to work well during evaluation?

CLIP’s pre-trained image encoder encodes robust visual representations learned from 400 million diverse image-text pairs. When you fine-tune that encoder on a narrow target domain — even with a lightweight adapter — you push those representations toward the specific visual features of your training data. The model becomes better at your training distribution and worse at everything outside it. This is the classic accuracy-robustness trade-off in domain adaptation, and it is especially consequential for CLIP because CLIP’s out-of-distribution generalization is the primary reason it became ubiquitous in production. Corrupting that generalization through aggressive image-side adaptation is counterproductive for any deployment environment that encounters varied real-world inputs.

How should a practitioner choose how many experts and what rank to use in A₃B₂?

The paper’s sensitivity analysis on ImageNet base-to-novel tasks found n=3 experts and rank r=32 to be optimal defaults. Too few experts limit model expressiveness; too many risk overfitting. The rank parameter faces the same trade-off as LoRA’s rank choice: lower ranks are more parameter-efficient but less expressive. The suppression strength λ_bias=0.3 provides moderate uncertainty-aware dampening — too high over-constrains visual adaptation; too low leaves the Branch Bias uncorrected. For most few-shot tasks in the 1–16 shot range, the paper’s default hyperparameters are a reliable starting point before task-specific tuning.

What happens to A₃B₂ when the model is confidently wrong about an out-of-distribution input?

This is an important limitation the paper does not fully resolve. The UAAD mechanism uses maximum softmax probability as its confidence proxy, a heuristic that is known in the uncertainty quantification literature to be overconfident on some out-of-distribution inputs — the model may assign high probability to an incorrect class for an unfamiliar input, bypassing the dampening mechanism precisely when it is needed. The paper demonstrates strong results on standard benchmarks but does not evaluate the method in adversarial out-of-distribution regimes where softmax overconfidence is most severe. Future work using better-calibrated confidence estimators — temperature scaling, ensemble methods, or Bayesian approaches — could address this gap.