Adding a quality indicator or metric¶
Use a quality indicator for a numerical property of an objective front. Execution counters and timing belong to experiment outcomes; analysis-only derived columns belong to UX/study reporting rather than the kernel interface.
Workflow¶
- Put front indicators under
src/vamos/foundation/quality_indicators/. Implement theQualityIndicatorprotocol when the metric fits the commoncompute(front, reference_front, maximise, **kwargs)contract and returnIndicatorResult. - Validate that fronts are finite two-dimensional arrays and that reference fronts/points match objective dimension. Define minimization/maximization semantics explicitly.
- Add the canonical name and any essential spelling to the alias table behind
get_indicator(...)andcanonical_indicator_name(...); export intentional classes from the package__init__.py. Do not create a parallel registry. - Keep optional MooCore use guarded. If a NumPy implementation exists, test it as the reference; otherwise raise the established actionable optional-dependency error.
- Add a
KernelBackendhook only for a high-call-count primitive that algorithms need directly. Advertise it throughquality_indicators()and test capability before dispatch. - Update study/UX choices only when the indicator is supported there, and document its reference data, direction, units, and failure behavior.
Indicator contract¶
VAMOS 1.2.0: every indicator path validates its inputs the same way.
- Finite inputs: a NaN or infinite value in a front, a reference front or a reference point raises
ValueError. MooCore used to drop such rows silently. - Reference point:
hypervolume(),hypervolume_contributions(),HVIndicator,HVContributionsIndicator,HVApproxIndicatorandWHVRectIndicatorrequire an explicit reference point. VAMOS 1.1.0 usedmax(front) + 1of the measured front, so adding a dominated point changed the hypervolume. - Points outside the reference box: every path ignores them, as MooCore does.
hypervolume()no longer raises for them, and itsallow_ref_expandargument is ignored with aFutureWarning. - Contributions: they follow MooCore's
ignore_dominatedsemantics in every environment. A dominated, duplicated or outside point contributes 0. Without MooCore, the fallback for three or more objectives used to keep dominated points in, and could return small negative values. pareto_filter: it rejects rows with NaN and inputs that are not two-dimensional.- Study and benchmark indicators: they accept every alias of
get_indicator. The hypervolume family receives the problem's reference point, and the other indicators receive its reference front.
Definitions to keep in mind when comparing with other tools:
- IGD:
igdis the mean distance from each reference point to the front (p = 1, Coello and Cortés, as in pymoo). jMetal's default IGD issqrt(sum d^2) / n, which gives smaller values. - Multiplicative epsilon:
epsilon_multneeds strictly positive values in the front and in the reference front. Shift both explicitly when objectives can be 0, as on ZDT and DTLZ fronts. - Exact fallback cost: without MooCore, the exact hypervolume for four or more objectives grows exponentially with the number of objectives. SMS-EMOA and IBEA with the hypervolume indicator need the
[compute]extra in practice from M = 4. - Approximate hypervolume: Development after 1.4.0:
HVIndicatorandHVApproxIndicator(hv_approx) are exported by the experimentalvamos.ux.apiand documented in Analysis and Visualization.hv_approxpassessamples,seedandmethod(one ofHV_APPROX_METHODS;Noneis MooCore's default) tomoocore.hv_approxand records them indetails; an unknown method raisesValueError.
Do not persist a second metric file beside a canonical run. Bounded scalar metrics that belong to a run go in the manifest outcome; study tables are derived exports.
Required tests¶
- Known analytical examples and invalid shapes/reference dimensions.
- Optional-dependency behavior with and without the provider.
- Alias/name resolution through
get_indicator. - Backend parity when adding a native hook.
- Study integration only through canonical RunManifest metrics projected by
StudySummary.
Run:
python -m pytest -q tests/foundation/test_moocore_indicators.py tests/foundation/test_hypervolume_fallback.py
path: src/vamos/foundation/quality_indicators
path: src/vamos/foundation/quality_indicators/moocore_indicators.py
path: src/vamos/foundation/kernel/backend.py
path: tests/foundation/test_moocore_indicators.py
path: tests/foundation/test_hypervolume_fallback.py
symbol: vamos.foundation.quality_indicators.moocore_indicators:QualityIndicator
symbol: vamos.foundation.quality_indicators.moocore_indicators:IndicatorResult
symbol: vamos.foundation.quality_indicators.moocore_indicators:get_indicator
command: python -m pytest -q tests/foundation/test_moocore_indicators.py tests/foundation/test_hypervolume_fallback.py