Huanran Chen

Generalization is my central problem in machine learning. In deep learning, I believe it is deeply entangled with optimization, rather than cleanly separable through the traditional bias-variance picture.

I have been greatly inspired by previous work on the relationship between sharpness and generalization. In recent years, I identified another implicit bias—which I term closeness—that strongly correlates with generalization across various settings.

The intuition for “closeness” struck me one day while visualizing the loss landscapes of multiple tasks. Take LLM pre-training, for example: the model optimizes an averaged loss across diverse data sources. However, I realized that achieving a low total training loss isn’t the whole story. As illustrated in the figure above, two completely different types of minimizers can yield the exact same training loss. But a minimizer that lies geometrically “close” to all individual task minima (the intersection) inherently captures the shared underlying structure, leading to significantly better downstream generalization than a distant minimizer (the sum).

I further discovered that this geometric closeness is strictly upper-bounded by gradient closeness. Intuitively, if the gradients of each loss always share the same direction, their respective minimizers must be exactly the same. Moreover, gradient closeness serves as a powerful proxy for generalization, as demonstrated by the first-order approximation:

Thus, when I mention “closeness”, it encompasses both geometric and gradient closeness. To explicitly optimize this, I proposed closeness-aware second-order optimizers (such as CWA and its recent evolution, Nexus).

This optimization perspective produces consistent results across several settings:

1. LLM Pre-training: It achieves the exact same pretraining loss curve as the baseline, but delivers significantly better downstream task generalization.

Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima

Huanran Chen, Huaqing Zhang, Xiao Li, Yinpeng Dong, Ke Shen, Jun Zhu

• Paper

2. Black-box Transfer Attacks: It achieves much higher attack success rates than baselines. In this adversarial setting, models always trivially achieve 0 training loss, meaning optimization speed is irrelevant—generalization is all that matters.

Rethinking Model Ensemble in Transfer-based Adversarial Attacks

Huanran Chen, Yichi Zhang, Yinpeng Dong, Jun Zhu

• ICLR 2024 · Blog · Video

3. Breaking Production VLMs: By maintaining closeness, the optimizer successfully bypassed the defenses of commercial black-box models like Google Gemini, GPT-4V, and Bard with a >95% attack success rate.

How Robust is Google’s Bard to Adversarial Image Attacks?

Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, Jun Zhu

• NeurIPS Workshop 2023 · Blog · Video

While plotting empirical loss landscapes of large language models, I observed that they often settle into broad, flat “basins” rather than isolated minima. This geometry connects to how models preserve knowledge and resist perturbations.

Unveiling the Basin-Like Loss Landscape in Large Language Models

Huanran Chen, Zeming Wei, Yao Huang, Yichi Zhang, Yinpeng Dong, Jun Zhu.

• The Basin Phenomenon: We discovered that LLMs exhibit remarkable parameter-space robustness, forming stable “basins” across diverse benchmarks (e.g., Math, Coding, Safety) and models (Llama 2/3, Mistral, Vicuna). This basin phenomenon is non-trivial: it gradually emerges during the training process and consistently widens as model capacity scales up from 0.5B to 14B parameters. Interestingly, anywhere within this basin, the model generates diverse but consistently correct (or safe) responses.
• Implications for Fine-Tuning & Attacks: By visualizing the loss landscape along the Supervised Fine-Tuning (SFT) direction, we provide a clear geometric explanation for both catastrophic forgetting and fine-tuning attacks (e.g., performing normally on standard tasks but generating harmful content on malicious queries).
• Theoretical Grounding via Randomized Smoothing: We mathematically proved that average-case robustness to Gaussian noise inherently bounds the performance degradation caused by worst-case perturbations, including SFT. Furthermore, we elegantly extended this theoretical guarantee to any known noise distribution by reducing the certified bound calculation to a fractional or 0-1 knapsack problem.

• Pre-training Regularization: We demonstrated that incorporating Gaussian noise during the pre-training phase effectively transfers to enhanced robustness during downstream SFT, offering a principled way to widen these safe basins.

• ICLR 2026 · Blog

My first research project sparked my interest in the geometry of neural networks and the connection between optimization and generalization.

Bootstrap Generalization Ability from Loss Landscape Perspective

Huanran Chen, Shitong Shao, Ziyi Wang, Zirui Shang, Jin Chen, Xiaofeng Ji, Xinxiao Wu

• Proposed a novel backbone, scheduler, and training paradigm specifically designed to encourage the flatness of local optima during optimization.
• Provided a unified explanation for various empirical tricks and regularizations strictly through the lens of loss landscapes.

• Practical Recommendation: Demonstrated that our ALRS method provides a fundamental boost to standard SGD, offering a highly recommended plug-and-play solution if you are heavily relying on vanilla SGD.

• ECCV Workshop 2022 · Blog · Video

This line of work began during my undergraduate years. While studying diffusion models, I was struck by a fascinating counter-intuition: a purely generative model, trained solely to denoise, inherently contains all the conditional distribution information needed to perform discriminative tasks—without a single step of standard discriminative training.

Robust Classification via a Single Diffusion Model

Huanran Chen, Yinpeng Dong, Zhengyi Wang, Jun Zhu

• Paradigm Shift: Proposed directly utilizing a single diffusion generative model as a robust discriminative classifier, completely bypassing adversarial training.
• Empirical Validation: Demonstrated the inherent robustness of the diffusion classifier against both out-of-distribution (O.O.D.) data and adversarial examples.
• Theoretical Tools: Derived the optimal solution for diffusion models, providing a rigorous mathematical tool for analyzing their behavior.

• Efficient Inference: Proposed Likelihood Maximization as an efficient approximate marginal inference method.

• ICML 2024 · Blog · Video

The empirical robustness result motivated a stricter question: can we obtain a useful certified guarantee? The key was to bypass otherwise intractable components of the diffusion classifier.

Diffusion Models are Certifiably Robust Classifiers

Huanran Chen, Yinpeng Dong, Shitong Shao, Zhongkai Hao, Xiao Yang, Hang Su, Jun Zhu

• Derived two Evidence Lower Bounds (ELBOs) for log-likelihood on noisy data to construct new noised diffusion classifiers (enabling point-wise Lipschitz bounds).
• Obtained the analytical form of the gradient for diffusion classifiers, elegantly bypassing the intractable UNet Jacobian.
• Established three Lipschitzness and robustness lower bounds, providing rigorous point-wise certification.

• Achieved state-of-the-art certified robustness (e.g., 70.7% for an $\ell_2$ radius of 0.5), closing the gap to merely 20% from the empirical upper bound.

• NeurIPS 2024 · Blog 1 · Blog 2 · Video

Later, I realized this theoretical framework wasn’t limited to continuous image spaces or Gaussian noise. We could generalize the robustness guarantee to any discrete noise distribution in Large Language Models by reducing the randomized smoothing process to a classical knapsack problem.

Towards the Worst-case Robustness of Large Language Models

Huanran Chen, Zeming Wei, Yinpeng Dong, Hang Su, Jun Zhu.

• Rigorous White-Box Evaluation: Conducted white-box evaluations that strictly align tokens during optimization and inference. Existing LLM defenses, including adversarial training (AT), drop to 0% robustness under this evaluation.
• The Knapsack Reduction: To provide a lower bound on worst-case robustness, I focused on randomized smoothing. I reduced the calculation of certified robustness under any smoothing distribution to a fractal or 0-1 knapsack problem, providing a universal solver.
• Kernel Superiority: Theoretically proved that the uniform kernel (analogous to diffusion) strictly dominates the absorbing kernel (masked generation) in terms of robustness at the same accuracy level. Proved it is the best data-independent kernel without knowing data priors.
• Mapping the Ceiling: Derived a Bayesian error upper bound for the uniform kernel, mapping the theoretical ceiling of robust guarantees.
• Symmetric Lemma: Introduced a symmetric lemma to analyze the trading graph, enabling elegant proofs for properties like the relationship between certified robustness and vocabulary size.

• Results: Achieved a certified radius of 2.00 in $\ell_2$ (against a Bayesian upper bound of 2.10) and 6.57 in the suffix setting (where empirical research typically focuses on a suffix length of 20).

• SafeGenAI @ NeurIPS 2024 · Blog · Video