Skip to content

Calibration and Validation

Terms

Calibration fits model parameters using a training set. Validation scores the fitted model against a separate held-out set. Reporting fit error as prediction error is not valid.

The first implemented correction is:

[ \hat{t}=\max(0,\alpha t_{\mathrm{analytic}}+\beta). ]

Two or more distinct analytic predictions fit (\alpha) and (\beta) by ordinary least squares. One observation fits a scale through the origin and is only a preliminary calibration. A non-positive fitted scale is rejected.

1. Define the context of use

Before measuring, write down the decision the model will support. Examples:

  • comparing queue counts for one NIC revision and firmware;
  • estimating one accelerator kernel family over a stated tensor range;
  • modelling ORV3 polling latency for one controller and bus topology.

Do not begin with a claim such as "predicts all Cnuas hardware performance."

2. Record target identity

At minimum preserve:

Area Required details
Hardware Product, revision, serial-independent configuration
Firmware Version and settings
Clocks and power Frequency, governor, thermal and power policy
Links Generation, width, speed, MTU, encoding
Software OS, kernel, driver, runtime, compiler, emulator versions
Workload Packet, message, or tensor shape; batch; operation count
Contention Queue, flow, lane, rank, and concurrency counts
Measurement Instrument, repetitions, warm-up, statistic

Use this complete description as the calibration target string or associate it with a target manifest stored beside the profile.

3. Split observations before fitting

Use separate files:

training.json:

{
  "observations": [
    {"predicted_ns": 1000, "measured_ns": 1170, "label": "4 KiB, one queue"},
    {"predicted_ns": 2000, "measured_ns": 2270, "label": "8 KiB, one queue"},
    {"predicted_ns": 4000, "measured_ns": 4470, "label": "16 KiB, one queue"}
  ]
}

held-out.json:

{
  "observations": [
    {"predicted_ns": 1500, "measured_ns": 1725, "label": "6 KiB, one queue"},
    {"predicted_ns": 3000, "measured_ns": 3360, "label": "12 KiB, one queue"}
  ]
}

The labels are required experimental context even though the current parser accepts an empty label.

4. Fit

cnuas-timing calibrate cnuasnic \
  "target product/revision; firmware; 100G; MTU; one queue; software versions" \
  training.json > calibration.json

The output records:

{
  "component": "cnuasnic",
  "target": "target product/revision; firmware; 100G; MTU; one queue; software versions",
  "scale": 1.1,
  "offset_ns": 70.0,
  "observations": 3,
  "fit_rmse_ns": 0.0
}

Fit RMSE describes the training data only.

5. Validate held-out data

cnuas-timing validate calibration.json held-out.json

The report contains:

Metric Meaning
MAE Mean absolute duration error
RMSE Root mean squared error, more sensitive to large errors
MAPE Mean absolute percentage error
Maximum absolute error Worst held-out miss

Also report error by workload regime. One aggregate can hide a packet-rate failure at small messages or a bandwidth failure at large messages.

6. Decide whether the affine model is adequate

Inspect residuals against:

  • work size;
  • queue and flow count;
  • concurrency;
  • compute versus memory intensity;
  • temperature and clock state.

A pattern means the profile is missing a resource or the affine correction is too simple. Do not add more coefficients only to reduce training error. Add a physically interpretable stage, split the operating range, or use a specialist simulator.

7. Apply a calibration in Python

from cnuas_timing import Observation, fit, load_profile

profile = load_profile("profiles/cnuas-analytic-v0.json")
calibration = fit(
    "cnuasnic",
    "named target and workload conditions",
    [Observation(1000, 1170), Observation(2000, 2270)],
)
calibrated_nic = calibration.apply(profile.component("cnuasnic"))

Create a new TimingProfile with the calibrated component and use dump_profile to preserve it. Never overwrite the analytic reference profile.

Publication checklist

  • State the evidence class.
  • Publish profile and pipeline versions.
  • Publish raw observations when permitted.
  • Identify training and held-out rows.
  • Report measurement variability, not only model error.
  • Report the workload range.
  • Avoid extrapolation outside that range.
  • Keep the target name with every calibrated result.