From 67771c7cfef546e37fd3b4ea3f3382dc1c2e381c Mon Sep 17 00:00:00 2001 From: Prathamesh Mandke <46148373+pkmandke@users.noreply.github.com> Date: Tue, 15 Sep 2026 13:20:34 -0700 Subject: [PATCH 1/6] add bpw section Signed-off-by: Prathamesh Mandke <46148373+pkmandke@users.noreply.github.com> --- docs/src/utils/mixed_precision.md | 43 +++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/docs/src/utils/mixed_precision.md b/docs/src/utils/mixed_precision.md index 58b76c49..c46d123a 100644 --- a/docs/src/utils/mixed_precision.md +++ b/docs/src/utils/mixed_precision.md @@ -69,3 +69,46 @@ Likewise, the per-layer setting being varied across candidate configs does not h - [Mixed-precision palettization with ResNet50](../examples/model_examples/mixed_precision_palettization.md) — applies palettization with 2/4/6-bit per-tensor candidate configs and the greedy approach targeting a BPW of 4. - [coreai-models](https://github.com/apple/coreai-models) — the same workflow is applied to a few LLMs in this repository to produce mixed precision compression recipes. Users can find the mixed precision configs in the repo and apply them with `coreai-opt`. + +## Utility for computing analytical Bits Per Weight (BPW) + +{func}`~coreai_opt.inspection.bits_per_weight` is a utility that computes an analytical BPW estimate from a *prepared* `coreai-opt` model. + +It estimates the average bit width of a model, amortizing compression overhead such as quantization scales and zero-points as well as palettization look-up tables and per-channel scales. Compressed tensors are counted at their effective compressed cost and everything else (biases, norms and buffers such as BatchNorm running statistics) is counted at its full-precision dtype cost. + +**Usage:** + +```python +from coreai_opt.quantization import Quantizer, QuantizerConfig +from coreai_opt.quantization.config import ExecutionMode +from coreai_opt.inspection import bits_per_weight + +# Prepare an eager-mode weight-quantized model +quantizer = Quantizer( + model, QuantizerConfig.presets.w8(execution_mode=ExecutionMode.EAGER) +) +prepared_model = quantizer.prepare(example_inputs) + +result = bits_per_weight(prepared_model) +print(result.bpw) # e.g. 8.86 +``` + +The returned {class}`~coreai_opt.inspection.BitsPerWeightResult` also exposes `per_module_map`, a mapping from module name to the module's own average BPW, which is useful for inspecting how bits are distributed across different parts of the model. + +The utility supports: + +- **Eager-mode integer weight quantization**: int8, int4, int2 and their unsigned variants along with symmetric and asymmetric, at any granularity. +- **Eager-mode floating-point weight quantization**: FP8 (`e4m3`, `e5m2`) and FP4 (`e2m1`). +- **Palettization** at any spec-supported `n_bits`, including a quantized LUT. + +Sub-byte payloads and zero-points are packed at `n_bits` with no padding. + +:::{note} +Pruned models and graph-mode (`torch.fx.GraphModule`) quantized models are currently not supported. +::: + +:::{note} +This is an analytical estimate computed on a *prepared* `coreai-opt` model and as such does not reflect the exported asset size of any [`finalize()` backend](../introduction/integration_coreai.md). Passing a finalized model to the utility is not supported. +::: + +For the full API, see the {func}`~coreai_opt.inspection.bits_per_weight` reference. From 3938b517ebe310c8b05d60088763c9e3f156a565 Mon Sep 17 00:00:00 2001 From: Prathamesh Mandke <46148373+pkmandke@users.noreply.github.com> Date: Tue, 15 Sep 2026 16:11:39 -0700 Subject: [PATCH 2/6] cross ref Signed-off-by: Prathamesh Mandke <46148373+pkmandke@users.noreply.github.com> --- docs/src/quantization/config.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/src/quantization/config.md b/docs/src/quantization/config.md index 4f0f2d17..0a9104d4 100644 --- a/docs/src/quantization/config.md +++ b/docs/src/quantization/config.md @@ -240,6 +240,8 @@ quantization_config: This config would apply the FP4 quantization to all supported ops' weights, and FP8 quantization to all supported ops' activations at both inputs and outputs. (See discussion in [Two Execution Modes](overview.md#two-execution-modes-graph-and-eager) on the supported ops and patterns for quantization). +To estimate what a weight config like this costs in terms of bits per weight (BPW), use the {func}`~coreai_opt.inspection.bits_per_weight` utility to get an analytical BPW estimate for a *prepared* `coreai-opt` model. + `global_config` allows access to the scope of all supported ops and modules in the model. Three fields target specific tensor groups within a module's operation: From 6eb6bf3be8eb4afb77cf8adf42135c03ed2f7d0e Mon Sep 17 00:00:00 2001 From: Prathamesh Mandke <46148373+pkmandke@users.noreply.github.com> Date: Tue, 15 Sep 2026 16:56:10 -0700 Subject: [PATCH 3/6] move Signed-off-by: Prathamesh Mandke <46148373+pkmandke@users.noreply.github.com> --- docs/src/quantization/config.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/src/quantization/config.md b/docs/src/quantization/config.md index 0a9104d4..8b90bdf1 100644 --- a/docs/src/quantization/config.md +++ b/docs/src/quantization/config.md @@ -15,6 +15,8 @@ Regarding the terminology of ops and modules, as used in these APIs, here is wha {class}`~coreai_opt.quantization.config.QuantizerConfig` lets you customize quantization settings for different parts of the model, via either modules or ops, or a combination of both. +To estimate what a config costs in terms of bits per weight (BPW), use the {func}`~coreai_opt.inspection.bits_per_weight` utility to get an analytical BPW estimate for a *prepared* `coreai-opt` model. + We will first take a look at the {class}`~coreai_opt.quantization.spec.QuantizationSpec` class and then through examples, will walk over how to define the config classes (and thereby covering how they are structured). ## QuantizationSpec @@ -240,8 +242,6 @@ quantization_config: This config would apply the FP4 quantization to all supported ops' weights, and FP8 quantization to all supported ops' activations at both inputs and outputs. (See discussion in [Two Execution Modes](overview.md#two-execution-modes-graph-and-eager) on the supported ops and patterns for quantization). -To estimate what a weight config like this costs in terms of bits per weight (BPW), use the {func}`~coreai_opt.inspection.bits_per_weight` utility to get an analytical BPW estimate for a *prepared* `coreai-opt` model. - `global_config` allows access to the scope of all supported ops and modules in the model. Three fields target specific tensor groups within a module's operation: From 9595a8ee41b8e1aab1db469c1bc842fb363b0987 Mon Sep 17 00:00:00 2001 From: Prathamesh Mandke <46148373+pkmandke@users.noreply.github.com> Date: Fri, 18 Sep 2026 11:30:10 -0700 Subject: [PATCH 4/6] address comments Signed-off-by: Prathamesh Mandke <46148373+pkmandke@users.noreply.github.com> --- docs/src/utils/mixed_precision.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/src/utils/mixed_precision.md b/docs/src/utils/mixed_precision.md index c46d123a..7eeb7dfc 100644 --- a/docs/src/utils/mixed_precision.md +++ b/docs/src/utils/mixed_precision.md @@ -38,6 +38,10 @@ Once we have the sensitivity scores of all the layers and candidate configs, we Several strategies can be applied for determining the mixed-precision recipe given a constraint — for example, a target average bits-per-weight (BPW) — and aim to balance model size reduction with minimal accuracy loss. +:::{note} +To obtain the analytical average BPW of a model, see [Utility for computing analytical Bits Per Weight (BPW)](#utility-for-computing-analytical-bits-per-weight-bpw) below. +::: + A simple greedy approach often works well: 1. Sort all `(layer, config)` tuples by sensitivity in descending order (least quality loss first). @@ -74,14 +78,14 @@ Likewise, the per-layer setting being varied across candidate configs does not h {func}`~coreai_opt.inspection.bits_per_weight` is a utility that computes an analytical BPW estimate from a *prepared* `coreai-opt` model. -It estimates the average bit width of a model, amortizing compression overhead such as quantization scales and zero-points as well as palettization look-up tables and per-channel scales. Compressed tensors are counted at their effective compressed cost and everything else (biases, norms and buffers such as BatchNorm running statistics) is counted at its full-precision dtype cost. +It estimates the average bit width of a model, accounting for compression overhead such as quantization scales and zero-points as well as palettization look-up tables and per-channel scales. Compressed tensors are counted at their effective compressed cost and everything else (biases, norms and buffers such as BatchNorm running statistics) is counted at its full-precision dtype cost. **Usage:** ```python from coreai_opt.quantization import Quantizer, QuantizerConfig from coreai_opt.quantization.config import ExecutionMode -from coreai_opt.inspection import bits_per_weight +from coreai_opt.inspection import BitsPerWeightResult, bits_per_weight # Prepare an eager-mode weight-quantized model quantizer = Quantizer( @@ -89,7 +93,7 @@ quantizer = Quantizer( ) prepared_model = quantizer.prepare(example_inputs) -result = bits_per_weight(prepared_model) +result: BitsPerWeightResult = bits_per_weight(prepared_model) print(result.bpw) # e.g. 8.86 ``` @@ -97,14 +101,14 @@ The returned {class}`~coreai_opt.inspection.BitsPerWeightResult` also exposes `p The utility supports: -- **Eager-mode integer weight quantization**: int8, int4, int2 and their unsigned variants along with symmetric and asymmetric, at any granularity. +- **Eager-mode integer weight quantization**: any spec-supported integer dtype, any quantization scheme (symmetric or asymmetric), at any granularity. - **Eager-mode floating-point weight quantization**: FP8 (`e4m3`, `e5m2`) and FP4 (`e2m1`). - **Palettization** at any spec-supported `n_bits`, including a quantized LUT. Sub-byte payloads and zero-points are packed at `n_bits` with no padding. :::{note} -Pruned models and graph-mode (`torch.fx.GraphModule`) quantized models are currently not supported. +Sparsity (pruning) and models quantized using graph mode (`ExecutionMode.GRAPH`) are currently not supported. ::: :::{note} From 5a35aa89014c643b76511367f4f0e63c76647197 Mon Sep 17 00:00:00 2001 From: Prathamesh <46148373+pkmandke@users.noreply.github.com> Date: Fri, 18 Sep 2026 11:32:01 -0700 Subject: [PATCH 5/6] Update docs/src/quantization/config.md Co-authored-by: Qiaoyu (Joey) Deng --- docs/src/quantization/config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/quantization/config.md b/docs/src/quantization/config.md index 8b90bdf1..7d015c8a 100644 --- a/docs/src/quantization/config.md +++ b/docs/src/quantization/config.md @@ -15,7 +15,7 @@ Regarding the terminology of ops and modules, as used in these APIs, here is wha {class}`~coreai_opt.quantization.config.QuantizerConfig` lets you customize quantization settings for different parts of the model, via either modules or ops, or a combination of both. -To estimate what a config costs in terms of bits per weight (BPW), use the {func}`~coreai_opt.inspection.bits_per_weight` utility to get an analytical BPW estimate for a *prepared* `coreai-opt` model. +To estimate what a config costs in terms of bits per weight (BPW), use the {func}`~coreai_opt.inspection.bits_per_weight` utility to get an analytical BPW estimate for a model *prepared* by coreai-opt. We will first take a look at the {class}`~coreai_opt.quantization.spec.QuantizationSpec` class and then through examples, will walk over how to define the config classes (and thereby covering how they are structured). From a1e046b606bafb81b58298b068d929a80749e340 Mon Sep 17 00:00:00 2001 From: Prathamesh Mandke <46148373+pkmandke@users.noreply.github.com> Date: Fri, 18 Sep 2026 12:50:03 -0700 Subject: [PATCH 6/6] review comments Signed-off-by: Prathamesh Mandke <46148373+pkmandke@users.noreply.github.com> --- docs/src/palettization/config.md | 2 ++ docs/src/quantization/config.md | 2 +- docs/src/utils/mixed_precision.md | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/src/palettization/config.md b/docs/src/palettization/config.md index 90a88b2a..b494a936 100644 --- a/docs/src/palettization/config.md +++ b/docs/src/palettization/config.md @@ -4,6 +4,8 @@ Palettization Configs follow the same philosophy as the [Quantization Config](.. They are simpler as palettization applies only to the weights in the model. (Hence there are no `op_input_spec` and `op_output_spec` fields in the {class}`~coreai_opt.palettization.config.ModuleKMeansPalettizerConfig` and {class}`~coreai_opt.palettization.config.OpKMeansPalettizerConfig`.) +To estimate what a config costs in terms of bits per weight (BPW), use the {func}`~coreai_opt.inspection.bits_per_weight` utility to get an analytical BPW estimate for a *prepared* `coreai-opt` model. See [Utility for computing analytical Bits Per Weight (BPW)](../utils/mixed_precision.md#utility-for-computing-analytical-bits-per-weight-bpw) for details. + ## PalettizationSpec {class}`~coreai_opt.palettization.spec.PalettizationSpec` defines the following key properties, among others (for full list see API reference): diff --git a/docs/src/quantization/config.md b/docs/src/quantization/config.md index 7d015c8a..a884b6e7 100644 --- a/docs/src/quantization/config.md +++ b/docs/src/quantization/config.md @@ -15,7 +15,7 @@ Regarding the terminology of ops and modules, as used in these APIs, here is wha {class}`~coreai_opt.quantization.config.QuantizerConfig` lets you customize quantization settings for different parts of the model, via either modules or ops, or a combination of both. -To estimate what a config costs in terms of bits per weight (BPW), use the {func}`~coreai_opt.inspection.bits_per_weight` utility to get an analytical BPW estimate for a model *prepared* by coreai-opt. +To estimate what a config costs in terms of bits per weight (BPW), use the {func}`~coreai_opt.inspection.bits_per_weight` utility to get an analytical BPW estimate for a model *prepared* by `coreai-opt`. See [Utility for computing analytical Bits Per Weight (BPW)](../utils/mixed_precision.md#utility-for-computing-analytical-bits-per-weight-bpw) for details. We will first take a look at the {class}`~coreai_opt.quantization.spec.QuantizationSpec` class and then through examples, will walk over how to define the config classes (and thereby covering how they are structured). diff --git a/docs/src/utils/mixed_precision.md b/docs/src/utils/mixed_precision.md index 7eeb7dfc..ba73e900 100644 --- a/docs/src/utils/mixed_precision.md +++ b/docs/src/utils/mixed_precision.md @@ -78,7 +78,7 @@ Likewise, the per-layer setting being varied across candidate configs does not h {func}`~coreai_opt.inspection.bits_per_weight` is a utility that computes an analytical BPW estimate from a *prepared* `coreai-opt` model. -It estimates the average bit width of a model, accounting for compression overhead such as quantization scales and zero-points as well as palettization look-up tables and per-channel scales. Compressed tensors are counted at their effective compressed cost and everything else (biases, norms and buffers such as BatchNorm running statistics) is counted at its full-precision dtype cost. +It estimates the average bitwidth of a model, accounting for compression overhead such as quantization scales and zero-points as well as palettization look-up tables and per-channel scales. Compressed tensors are counted at their effective compressed cost and uncompressed tensors are counted at their full-precision dtype cost. **Usage:**