panoptes.pipeline.provenance¶
panoptes.pipeline.provenance ¶
Where a calibration value came from, recorded beside the value itself.
The old flow serialized a dump of global settings into every document, so
"saturation was 11435, read from WHTLVLN" and "saturation was 15872,
because nobody knew" came out identical. That is how
params_camera_saturation=15872 came to sit in archive records against 37
different cameras: the pipeline faithfully recorded a fleet-wide default and
nothing downstream could tell it apart from a measurement.
A Resolved carries the value and the tier it came from, so the two cases
are distinguishable by anything reading the archive. See data contract 3.4.
The tiers are ordered best to worst. Nothing here refuses to produce a value:
declaring which header keywords are required, and failing when one is missing,
is a separate change against the header contract. What this module guarantees
is that a fallback is never silent -- it is labeled Provenance.DEFAULT and
carries the name of the setting it fell back to.
One rule follows for anything reading the archive, and it is worth stating
where the tiers are defined: params_camera_* in existing records is
never a source for a camera profile. Those fields are the fleet-wide
defaults, so reading them back would close a loop in which the pipeline
rediscovers its own wrong constants.
Provenance ¶
Bases: StrEnum
Where a resolved value came from, best tier first.
StrEnum so the value serializes as a plain string with no encoder hook,
which is what keeps the document loadable into a document store.
Source code in src/panoptes/pipeline/provenance.py
Resolved ¶
Bases: BaseModel
A calibration value together with the tier it was resolved from.
source names the specific origin within the tier -- a header keyword
("WHTLVLN"), or the settings attribute a default came from
("camera.saturation") -- so a record says not just how well a value is
known but exactly where to look to check it.
Source code in src/panoptes/pipeline/provenance.py
from_header ¶
Resolve keyword from header, or return None if it is absent.
Returns None rather than a Resolved with a null value, so a caller can
write from_header(...) or fallback(...) and have the tiers fall through
in order. An empty string counts as absent: POCS writes SEQID = '' on
some frames, and an empty keyword is missing data rather than a value.
Source code in src/panoptes/pipeline/provenance.py
from_default ¶
Label value as a fleet-wide default that nobody could do better than.
resolve_camera ¶
Resolve the per-frame camera calibration, each value with its tier.
Four values, and they are known to very different degrees:
saturation
WHTLVLN gives the white level per frame, offline, on every frame
in the archive back to 2016. Measured values run 11435--12277 against
a fleet-wide default of 15872, so both cameras checked in data
contract 2.1 sit roughly 25% below the hardcoded threshold and the
default fails to mask saturated pixels on either. Free, and worth
taking.
image_width / image_height
NAXIS1/NAXIS2, which are in every FITS file by definition.
The old code read IMAGEW/IMAGEH, which POCS never writes --
astrometry.net adds them during plate solving, so they are present on
solved frames and absent on raw ones. int(header.get("IMAGEW", 0))
then yields a silent zero, which is why 16% of the observation index
has null dimensions.
effective_gain
EGAIN where POCS wrote it, which is rare. Otherwise a default,
until the camera registry can answer it by body serial.
zero_bias
Not in the header at all, on any frame, so this is always a default
today. It is load-bearing content for the registry.
Source code in src/panoptes/pipeline/provenance.py
as_document ¶
Render resolved values as plain nested maps for the metadata document.
defaulted ¶
Names of the values that fell back to a fleet-wide default.
What a caller warns about, and what a survey of the archive counts.