TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge
cs.LG, cs.AI, eess.SP
Submitted: 2026-09-27
Updated: 2026-09-27
Terminology
Sources
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- On Adaptivity in Zeroth-Order Optimization
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- On-Device Fine-Tuning via Backprop-Free Zeroth-Order Optimization
- Ternary Weight Networks
- Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
- BitNet b1.58 2B4T Technical Report
- Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations
- A White Paper on Neural Network Quantization
- Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
- BitNet: Scaling 1-bit Transformers for Large Language Models
- Magicoder: Empowering Code Generation with OSS-Instruct
- BitNet Distillation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks