Enhancements
Corrections and new science, in a separate installable package the port never depends on.
In short: arrayprobe never depends on this package, and a test fails if that reverses. Every correction ships with its effect measured both ways on the same data, including the ones measured inert.
Which engine? Enhancements act on the UIOOPS v3.5 engine; an earlier translation of UIOOPS v3.1.4 reproduced archived products and is not part of this package. Four modules correct defects the UIOOPS developers fixed themselves in v3.5, where applying one would double-correct. Each module declares its engines, enforced at resolution time: Which enhancements apply to which engine.
When does one act? Each module declares a stage (decode, image processing, size distribution and combine, or post-combine export); a proc-file correction has nothing to do until framing has run. The diode and inter-arrival tools, which are not corrections, declare one too: Which enhancements act at which processing-chain stage.
This page covers corrections against UIOOPS (arrayprobe-uiops-enhancements, 36 modules). SODA-2, NCAR's IDL chain, has its own, enhancements_soda2 (arrayprobe-soda2-enhancements) — 18 correction modules plus one diagnostics-only module, documented in enhancements_soda2/README.md and enhancements_soda2/BACKLOG_STATUS.md.
See also Findings for what is being corrected · Conventions for why the split exists · All-on sets, per probe and engine for what "every enhancement on" means for a given probe.
In a notebook: examples/2_processing/09_enhancement_ab.ipynb switches outoffocus on in an [enhancements] table and runs it A/B on the tracked dense demo flight under both engines, down to the size distribution.
An engine translates its original faithfully: there, a "fix" that changes output is a defect, except the default-on corrections to data-destroying defects listed in Choosing an engine. Everything that changes output lives in arrayprobe-uiops-enhancements, which depends on arrayprobe and never the reverse — enforced by an AST-based test and verified by uninstalling the package and re-running the port's full suite.
python -m pip install arrayprobe-uiops-enhancementsContributors working from a checkout of the repository install it in editable mode instead: .venv/Scripts/python.exe -m pip install -e enhancements_uiops (Linux / macOS: .venv/bin/python).
Each module is opt-in, has its own design document, and reports its effect on real data, including where it is zero: listing only confirmed problems would distort the picture.
Speed work is the exception: it lives in both spaces
Performance is split not by where it came from but by whether it moves a written byte.
| The change | Where it lives |
|---|---|
| Byte-identical, opt-in, no new dependency | the port |
| Changes any output, needs a dependency the port refuses, or trades accuracy for time | enhancements |
So the parallel decode and flags (--s2-jobs, --chunks, --jobs) are in the port: their output is byte-identical to the serial path, asserted on every channel selection and every sidecar, and all three default to serial (user guide). No module in this package is a speed enhancement.
| Module | What it addresses | Effect on real data | ||||
|---|---|---|---|---|---|---|
wide_hollow_fix.py | the hollow test packs diodes 1–32 whatever the probe's width, so a particle beyond diode 32 is ruled hollow on no evidence | 64 of 106 such calls (60.4 %) spurious, across ten flights | ||||
int16_double_fix.py | int16 × double collapses the particle diameter for four of six size definitions | real in MATLAB; measured inert on the validation campaign's output, for two independent reasons | ||||
hole_edge_swap.py | max_hole_diameter and edge_at_max_hole written to swapped variables | 845 of 49,637 particles on one flight; the physical invariant is reversed in all of them | ||||
conc_arear_fix.py | conc_AreaR scaled by diode size twice, zeroing the variable | 410 real non-zero cells recovered on one file | ||||
reshape_fix.py | mean_area_ratio written without its siblings' transpose — a column-major reshape, so values land at wrong coordinates | correcting reshape verified three independent ways | ||||
spot_edge_fix.py | spot_edge(147) is the literal 0757 — a missing decimal point | zero particles fall in the vulnerable band on six flights: a measured "fine" | ||||
pms_c6_routing_fix.py | Fast2DC scalars land in the 2DC output file | pure routing error; surfaced a second issue left as an open question | ||||
last_record_fix.py | the last record of every raw file is never decoded, so particles beginning in it are dropped | 0–235 particles per file across all 24 raw files — but 0–14 of them reach the used size bins, so it barely moves a distribution | ||||
zero_image_recovery.py | a modified copy of UIOOPS used by one campaign abandons a whole image record on a zero-image particle | 47 records abandoned in 2,000 records; 4,448 particles (+1.6 %) recovered, 119 of them in used bins (+1.57 %) — not size-biased, so it does move a distribution | ||||
overload_accounting_fix.py | the DOF-overload accounting is reset for seconds that accept nothing, so sum_IntArr/sample_vol claim a full second | 223 seconds corrected on one file; concentrations unaffected either way | ||||
outoffocus.py | the Korolev out-of-focus size correction is dead code in every MATLAB tree | diagnostics re-derived bit-for-bit against stored columns, then the factor applied | ||||
particle_time_int8_fix.py | on 2DC/2DP and Fast2DC, the per-particle time inherits the header's int8 type and saturates, so particle_time reads 127 for virtually every real record while the file's Time is correct | not yet measured on campaign data; on by default for those probes, which have no archived product (--protective off skips it) | ||||
mask_presence_fix.py | the shadow-diode mask is looked up by a constructed file name with an empty else, so a missing or misnamed mask leaves the run unmasked with no message, and the output is byte-identical to a run that never asked for masking. Mask files written by find_shadow_diodes.m use a different name pattern from the one imgProc_sm looks for, so this happens in practice. The check names the mask it found or stops the run | changes no number: the 2016-09-06 2D-S masks are empty, so masking is inert for that product; what changes is that a run states which mask it used. Off by default | ||||
particle_number_all_fix.py | on 2DC, 2DP and Fast2DC, imgProc_sm.m defines Particle_number_all under an ` | guard that is always true but writes it under an && guard, so those files carry a column of pure fill values described as a particle index. The fix drops the column, which is what changing | to && would produce, so a reader gets a missing variable instead of -2147483647` | not applicable to 2D-S/HVPS, which write the variable correctly (240 of 240 real values on a real proc file); not yet measured on real 2DC data. Off by default | ||
particle_time_glitch_fix.py | on CIP/PIP, particle times come from the particle header's clock, which can glitch (isolated readings of 0, 202 or 316363); sizeDist treats the first decrease as midnight and adds 24 h to every later particle. The UIOOPS v3.5 day-crossover guard is fooled by the all-ones header, so this drops the glitched particles instead | on the real 2016-09-06 CIP file the cascade moves most of the second session past the last airspeed second, so the size distribution is silently empty for the rest of the flight; with the fix it covers the flight. On by default for CIP/PIP, which have no archived product (--protective off skips it) | ||||
pressure_units_fix.py | single_vt.m documents its pressure input in hPa, but one campaign's drivers pass a raw 9e5, so fall speed vt and precipitation rate Prec_rate are computed at the wrong pressure. The fix reruns the size distribution at a stated hPa pressure | on two real field-campaign size-distribution files, every non-zero cell of vt and Prec_rate moves (505 of 505, and 706 of 706) by a factor of 1.75–6.05 and 1.75–4.97; all 23 other variables are byte-identical. The combined product carries neither variable, so it does not move (enhancements_uiops/docs/applicability_matrix.md). Off by default | ||||
rollover_fix.py | sizeDist adds 24 h only after the first midnight decrease in the per-second time, because MATLAB's a:b uses only the first element of a vector; a flight crossing midnight more than once is mis-timed after the second crossing. The fix corrects every crossing | not applicable to the six validated flights, none of which crosses midnight (zero decreases); not yet measured on real data that does. Off by default | ||||
tas_grid_fix.py | when the airspeed file repeats a second, sizeDist loses every particle in that second. The fix collapses duplicated seconds, or uses a regular airspeed grid when an airspeed source is given | recovers 0.34–5.89 % of used-bin particles across four real files; it lowers agreement with the archived product (99.76 % → 96.90 % used-bin on one flight) because that product carries the defect too (enhancements_uiops/docs/applicability_matrix.md). On by default for 2DC/2DP, CIP/PIP and Fast2DC, which have no archived product (--protective off skips it); never applied where a 2D-S or HVPS archived product is being reproduced | ||||
poisson_restore.py | (literature method) the inter-arrival shattering test also rejects intact particles that happen to arrive close together. Following Korolev & Field (2015, Eq. 1–2), it restores them statistically from the expected Poisson spacing | derived on real inputs, not a production result: on the densest second of a 6-second field-campaign 2D-S file, τ = 6.47 µs against the 6 µs threshold gives a false-rejection probability of 0.170 and a 20.5 % concentration restoration (enhancements_uiops/docs/applicability_matrix.md). Off by default | ||||
partial_image_habit.py | (analysis) UIOOPS runs complete-image habit rules on edge-touching images or withholds a habit for them. Korolev & Sussman (2000, Sect. 4b) retrieve habit fractions of a population using accepted partial images too, and this implements that method; it writes no per-particle habit | changes no written value; the only published retrieval matrices are the OAP-2DC's, which the paper says do not transfer (p. 1057), so it refuses 2D-S and HVPS populations until a probe-specific matrix exists. Off by default | ||||
reaccept_every_event_fix.py | the reacceptance loop in sizeDist scans only the first shattering event, for the same first-element cause as the rollover, so later events never reaccept particles. The fix scans every event | inert on the validation campaign's configuration, which disables reacceptance; on a two-event test case run under Octave, the reaccepted index moves from 3 to 6, matching UIOOPS v3.5's own probetype-3 run; not yet measured on real data. Off by default ([enhancements] reaccept_every_event_fix = true turns it on) | ||||
published_dof.py | (literature method, 2D-S only) replaces the depth of field with the published 2D-S value of Gurganus & Lawson (2018, Eq. 1) where a bin's depth of field is below the arm distance | on 6 s of field-campaign 2D-S data, the two smallest bins (10 and 20 µm) of every volume-normalised variable rise by 0.676 %; capped bins are unchanged (enhancements_uiops/docs/applicability_matrix.md). Off by default ([enhancements] published_dof = true turns it on) | ||||
diode_range_fix.py | a shadow-diode mask entry above the probe's diode count silently widens the image, and imgProc_sm then exits part-way through the file with a success status, keeping only the particles written so far (a 128-diode mask applied to a 64-diode probe does this on the first particle). The fix is the obvious check: refuse the mask, naming every out-of-range entry and the probe width, before any particle is processed | a guard, not a recovery: no archived product is known to have been truncated (the masks held for the 2016-09-06 flight are empty or in range), and how often this has happened in the field is not estimated; a truncated file cannot be repaired, only re-run | ||||
distance_rejection.py | the shattering test rejects particle pairs closer than a fixed inter-arrival time, which assumes a fixed true airspeed: at high airspeed the same fragment spacing passes faster, so a fixed time threshold over-rejects at high speed and under-rejects at low speed. Following Korolev & Field (2015, AMT 8, 761–777, Sect. 4), it converts the threshold to a distance with the true airspeed and tests separation instead | impact not yet measured on real data; whether any held flight's airspeed varies enough within one shattering-fit block to matter has not been checked | ||||
brown_francis_1995.py | (mass–diameter relation) adds the Brown & Francis (1995, JTECH 12, 410–414, Eq. 3) power law M = 7.38×10⁻¹¹ D^1.9 (grams, D in microns) beside the mass–diameter relations already offered; the coefficients come from Locatelli & Hobbs (1974) aggregates of unrimed bullets, columns and side-planes. Its diameter is in microns, unlike the port's centimetre-based mass routine, and the module keeps the unit explicit | a choice of relation rather than a correction; its effect on ice water content from real data is not yet measured | ||||
cf_export.py | (export) the size-distribution file carries UIOOPS-native unit strings, no Conventions, no standard_name and an HHMMSS time axis, so CF-aware tools cannot label its axes or trust its units — and four of its unit strings do not describe the value they label (for example vt, labelled g/cm4). It writes a separate CF-1.10 copy with SI units keyed by variable, recording each rescale factor so the export is invertible; the original file is opened read-only | no number changes; each unit conversion is checked against the values it labels (for example, the per-habit mass sums to the total mass) | ||||
arm_export.py | (export) ARM publishes 2D-S/HVPS data, but ARM's Python toolkit (ACT) has no reader for any optical array probe. This writes a copy of a size-distribution or combined file whose time axis and global attributes follow the ARM Data File Standards (DOE/SC-ARM-15-004, v1.3), so ACT and xarray open it as an ARM user expects | no number changes; the copy is opened with ACT's own read_arm_netcdf in the module's tests. Site and facility codes are left to the caller, never defaulted | ||||
arco_export.py | (export) the engines write NETCDF3_CLASSIC, contiguous and uncompressed, to match MATLAB byte for byte, so reading one second means seeking through a whole variable. This writes an analysis-ready netCDF-4 copy with narrowed integer types (checked lossless per variable), chunking, and zlib compression, choosing the shuffle filter per variable from the data | lossless by default. Measured per variable in the module's documentation: shuffle helps integer data (8.6 % of baseline size with it, 10.7 % without) but hurts floating-point data (46.4 % with, 24.9 % without); narrowing floats to single precision would save only 1.4 points more (23.5 % against 24.9 %), so it stays off unless asked for | ||||
dmax_sensitivity.py | (diagnostic) UIOOPS offers six definitions of particle diameter, and a size distribution depends on which one a run picks, but nothing reports how far they disagree. This reports, per particle, the spread across all six on the same data, and flags the preserved int16 diameter saturation where it occurs | measured on 99 real particles from a tracked test file: mean relative spread across the six definitions 227.95 %, maximum 351.47 %, mean absolute spread 0.388 mm (enhancements_uiops/docs/applicability_matrix.md) | ||||
field2006_shatter.py | an alternative shattering correction: the Field et al. (2006) per-second fit of inter-arrival times, transcribed from NCAR's Process2D (process2d.cpp, Apache-2.0). Each second it fits a double Poisson to the last 400 inter-arrival times, sets a cutoff, rejects particles below it and scales concentrations by the fit's correction factor, writing these as extra variables beside the unchanged originals | impact not yet measured on real data; it is tested on synthetic files, and a value-for-value comparison against compiled Process2D is still to be made | ||||
int_arr_rollback_fix.py | on CIP/PIP probes, sizeDist has no branch to correct a backwards jump in the particle clock (the 2D-S/HVPS and 2DC branches exist), so the particle at the jump is rejected. The probe's clock is a time of day, so the jump it can suffer is midnight (86,400 s), not the counter span the other probes use; the module offers both magnitudes and makes the caller choose | on a real 12-record PIP file with one 86,400 s jump injected, MATLAB (run under Octave) and the faithful port both lose exactly the particle at the jump (584 → 583 counted) and the correction recovers it; how often real CIP/PIP data contains such a jump is not yet measured | ||||
intarr_threshold_fix.py | when deriving a shattering threshold from the inter-arrival histogram, IntArrAnalysis_time.m takes the global minimum of the counts inside the fitted window; a wide window includes the sparse tail, where the global minimum can land far from the shattering/real-particle dip. The fix takes the first interior local minimum of the same histogram and window, and reports how many candidates there were | on one real 2016 2D-S file's histogram the global minimum sits at 2.745 ms (4 counts) and the dip at 1.395 µs (94 counts), three orders of magnitude apart; no archived product of the validation campaign is affected, because those runs used the 6 µs default instead of a computed threshold, and the downstream effect on particle counts is not yet measured | ||||
intarr_policy.py | (choice of rule) McFarquhar et al. (2017, Meteor. Monogr. 58, ch. 11, p. 11.21) define the shattering threshold three ways and note the choice "can have a big impact on calculated N(Dmax)"; UIOOPS already computes all three but writes only one. This selects which of the three goes into the threshold file sizeDist reads; off reproduces UIOOPS | impact not yet measured on real data; on the synthetic dense demo flight, built with two designed gap values, the rules place the threshold on different sides of the designed gaps (enhancements_uiops/docs/CORRECTIONS.md, part intarr_policy.py) | ||||
poisson_uncertainty.py | (diagnostic) nothing in UIOOPS reports an uncertainty on a written concentration, so a bin resting on one particle and a bin resting on a thousand look alike. It adds per-bin and total Poisson counting uncertainty (relative uncertainty 1/√N for N particles), following Finlon et al. (2019, ACP 19) and the "100 particles in a size bin" significance convention quoted by McFarquhar et al. (2017, Meteor. Monogr. 58, ch. 11) | adds arrays, changes no existing value. On a 6-second size-distribution test file (768 cells), 4 cells are non-zero, none reaches 100 particles, and each carries a 100 % relative uncertainty because each rests on one particle (enhancements_uiops/docs/applicability_matrix.md). Off by default; opt in by name | ||||
probe_uncertainty.py | (reference) nothing in the output says how far to trust a size, a concentration or a shape class. It ships the published uncertainty ranges of Baumgardner et al. (2017, Meteor. Monogr. 58, ch. 9, Table 9-2) unnarrowed, the 2D-S sizing envelope of Vaillant de Guélis et al. (2019, AMT 12), and the shape floor the table's 5–12 pixels implies for a given diode size | changes no existing value. The shape floor is 50–120 µm for a 10 µm 2D-S and 750–1800 µm for a 150 µm HVPS; the 2D-S envelope returns ±10 % at 1000 µm and no value at 100, 200 and 400 µm, outside the measured bands (enhancements_uiops/docs/applicability_matrix.md). Off by default; opt in by name | ||||
effective_density.py | (mass–diameter relation) adds a third, independent mass path beside UIOOPS's own habit-keyed power law and the area-based Baker & Lawson relation: mass as density times volume, from a published habit and area-ratio density table | changes no existing value. On 99 real particles of a test file, 25 (25.3 %) map — aggregates whose area ratio lies in the table's range; the other 74 carry habit codes with no defensible match to the table's crystal types, so the module returns no value for them rather than guess (enhancements_uiops/docs/applicability_matrix.md). Off by default; opt in by name | ||||
md_relations.py | (mass–diameter relation) a registry of mass–diameter relations, each with its source, replacing hard-coded choices. It holds the four relations this package has verified against their sources (UIOOPS's single_mass, Baker & Lawson, the effective-density table, Brown & Francis 1995) and lists other published relations as declared gaps without coefficients, because those have not been read against their papers | a table, not a computation: changes no number by itself (enhancements_uiops/docs/applicability_matrix.md). Off by default; called by name | ||||
probe_crossover.py | (diagnostic, 2D-S + HVPS) no UIOOPS release merges two probes, and Schima et al. (2024, Sect. 4.2) found very weak 2D-S/HVPS correlation over the overlap for most habits and recommend a crossover diagnostic rather than an unguarded merge. It reports, second by second, whether the two probes' number concentrations agree over their shared size range, against counting statistics; it merges nothing | changes no computed value. On one field campaign's archived products, 42 probe pairs on 26 flights; of the 29 pairs with at least 10 effective particles on both probes from 0.375 mm up, the pooled 2D-S/HVPS ratio falls from a median 75 (bins 1–2) to 8.4 (bins 3 and up), and 3 of 29 pairs agree within counting statistics (enhancements_uiops/docs/applicability_matrix.md). Off by default; opt in by name | ||||
spec_artefact_filter.py | (2D-S only, selectable second rejection scheme) applies the 2D-S manufacturer's own artefact-removal criteria (SPEC 2D-S Post Processing Software Manual rev 1.1, 2011, Appendix B: roundness, splash, line-plus-dot noise, noisy-diode exceptions) as an independent accept/reject verdict beside UIOOPS's own reject codes, which it never changes; the manual's fifth, inter-event-distance test is left out | not yet measured on real data: it needs the raw image buffers, which the tracked test files do not carry (enhancements_uiops/docs/applicability_matrix.md). Off by default; opt in by name | ||||
particle_clustering.py | (tooling) k-means over shape/size fields with medoid representatives | feeds the triage server's cluster route |
SODA-2 corrections
SODA-2 is NCAR's IDL processing chain (3-clause BSD, UCAR/NCAR; cite Aaron Bansemer. (2026). abansemer/soda2: v2.26.0 (Version v2.26.0) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.18331418). Its corrections ship in arrayprobe-soda2-enhancements (enhancements_soda2/), one design record per module in enhancements_soda2/docs/CORRECTIONS.md. Every module is off by default except five that correct defects destroying or corrupting data (registry.PROTECTIVE), which run by default on every probe, because SODA-2 has written no archived product that must stay reproducible; --protective off skips them.
| Module | What it addresses | Effect on real data |
|---|---|---|
aspr_nan_fix.py | aspect_ratio.pro:105 clamps with IDL's MIN, which returns the finite operand for a non-finite aspect ratio, so a circle fit that never converged reports 1.0, indistinguishable from a round particle. The fix returns NaN instead | not yet measured on real data; on a synthetic circle with a NaN diameter the faithful path gives 1.0 and the fix NaN. Off by default |
den_clamp_fix.py | aspect_ratio.pro:66 clamps the divisor to at least 1 to guard a 0/0, but it also fires for any real circle with a radius below one pixel, understating the aspect ratio with no warning. The fix clamps only an exact zero | not yet measured on real data; on the finding's own test case the aspect ratio goes from 0.6136363636363636 to 0.7272727272727273, a 16% understatement corrected. Off by default |
rollover_modulus_fix.py | the 2D-S rollover constant at soda2_processbuffer.pro:318 is 2³²−1, one tick short of the 2³² the slice counter wraps at, so each wrap shifts the inter-arrival time by one tick | not yet measured on real data; on a wrap test case only the wrapping frames change, each by exactly one tick. Off by default |
label_blobs_int_count.py | label_blobs.pro:29 returns labels as bytes, so blob 256 becomes background and blob 257 aliases blob 1. The fix uses SODA-2's own integer-label option | not yet measured on real data; on a synthetic 400-blob image exactly one pixel is recovered. Off by default |
dilation_clamp_fix.py | label_blobs.pro:20 clamps the dilation kernel against the image's slice count rather than the padded array, so a 1- or 2-slice image gets a 3-wide kernel instead of 5, leaving fragments of one particle apart | not yet measured on real data; on a synthetic 1-slice image two fragments 3 diodes apart merge as intended. On by default |
fillholes_int_labels.py | fillholes.pro:9 labels holes as bytes, so an image with more than 255 hole regions loses one hole's fill | not yet measured on real data. Off by default |
poisson_fill_guard.py | poisson_spot_correct.pro:26-28 divides 0/0 or takes the square root of a negative when the filled area does not exceed the original; the result falls back to the no-spot default by accident. The fix guards the division and returns the defaults directly | on a 600 s window of a 2016 HVPS-3 raw file, all 32 compared output fields were exactly equal with the fix off and on; 190 of 191 particles reaching the routine took the guarded path (enhancements_soda2/docs/CORRECTIONS.md, part poisson_fill_guard.py). Off by default |
no_stuck_diode_fix.py | fixstuckbits.pro:17-20: when no diode is stuck, IDL's where() returns −1, which minus the padding is a valid index, so diode column n−8 is zeroed and reconstructed in real images whenever stuck-bit removal is on. The fix leaves the image unchanged when no diode crosses the threshold | not yet measured on real data. On by default |
blank_roi_dof_fix.py | binary_dof.pro:21-31 accepts an all-blank image as within depth of field, because the test reduces to 0 ge 0. The fix sets depth of field to 0 when the area is 0 | not yet measured on real data. Off by default |
empty_roi_diodegaps_fix.py | soda2_processbuffer.pro:707-709 reports a diode gap of 1 for an image with no shaded diodes, where 0 is expected. The fix returns 0 | not yet measured on real data. Off by default |
zero_area_return_fix.py | soda2_findsize.pro:9-13 returns its default structure for a zero-area particle, which says "centre in", a claim never checked, and that feeds the acceptance test. The fix sets centre-in to 0 for zero area | not yet measured on real data. Off by default |
last_buffer_index_fix.py | spec_index.pro:71 loops to numbuffs-2, so the particles in every file's last buffer are never indexed. The fix scans the last buffer too | not yet measured on real data. On by default |
same_time_buffer_fix.py | soda2_processbuffer.pro:362-365: when consecutive buffers carry the same time stamp, the first particle's inter-arrival time is never added, and the shortfall carries onto every later particle's time. The fix shifts those times by the missing interval | not yet measured on real data. On by default |
rejection_tally_fix.py | soda2_particlesort.pro:177-178 tallies only the first seven reject bits, so depth-of-field rejects land in the wrong column, and counts only the first matching reason. The fix returns a separate eight-column tally counting every reason, leaving SODA-2's own tally untouched | not yet measured on real data. Off by default |
hk_probeid_columns.py | spec_process_hk.pro:84-94 gives any probe id other than 'V', including a misspelling, the horizontal channel's housekeeping columns with no error. The fix refuses any id other than the three ids SODA-2 defines | not yet measured on real data. Off by default |
run_diagnostics.py | (diagnostics only) reports what SODA-2 drops without a count: frames dropped in a buffer, particles whose time index lands exactly on the last record, and two driver-level silent discards | changes no value; reports counts. Off by default |
pbp_inttime_consistent.py | soda2_particlesort.pro:198 vs :279-298: a negative inter-arrival time is written as 0 in the netCDF particle file and as the raw negative value in the CSV. The fix writes the clamped value to both | not yet measured on real data. Off by default |
pbp_rejectionflag_ubyte.py | soda2_process_2d.pro:236/:277 declare the particle file's rejectionflag a signed byte, so the depth-of-field bit (128) reads back as −128, while the CSV shows 128. The fix declares it unsigned | not yet measured on real data. On by default |
double_pi_fix.py | min_circle_fast.pro:34, :37, :41, :51 use single-precision pi, so an exact right angle fails the obtuse test and rectangles and disc outlines take the 3-point circle branch; the circle is unchanged but the corner count, and with it orientation and aspect ratio, can change | not yet measured on real data. Off by default |
Does it change your numbers?
A correct fix can have nothing to do on the configuration that produced the archived products. Three modules (wide_hollow_fix, spot_edge_fix, hole_edge_swap) are inert there — the modified reject thresholds skip the hollow block they correct — so their headline figures are stock-configuration measurements. enhancements_uiops/docs/applicability_matrix.md states this per module, measured on a real file.
An active module can still barely matter, because the first five size bins are unused. last_record_fix recovers hundreds of particles per file, single digits of them in used bins; zero_image_recovery recovers particles across the whole size range.
A figure measured on a sparse file may not carry to a dense one; only running both tells. outoffocus was measured in 2026 on a 49,637-particle file and again on the densest held, 17,289,449 particles, and the diameter-ratio and rate figures from that run still stand. The distribution-effect figure does not: outoffocus.py applied Korolev's (2007) correction in the wrong direction (it multiplied by size_factor where the paper divides), so the −6.1 % of used-bin particles and "every one of the 22 affected bins loses particles and none gains" result were both measured under that wrong-sign correction and are superseded (enhancements_uiops/docs/CORRECTIONS.md, part outoffocus.py, section The direction). The direction is fixed in code and covered by a regression test; no dense-file figure is quoted for the corrected direction.
Running them
arrayprobe run --enhancements <config.toml> <raw...> runs the engine the config names as a subprocess, resolves the enhancement set the config names, applies the ones with a driver adapter (12 of the 41, the keys of pipeline._APPLIERS) onto copies named <stem>.enh.<kind>.nc, and writes a <sd>.enhancements.json sidecar recording, per module, whether it resolved on, was applied, or was skipped and why. The port's own output is never touched. Which enhancements apply on which engine: Enhancements by engine.
How a correction is proved
Where the MATLAB computed something and mishandled it, there is a real oracle: the port's own validated value, or the stock tree under Octave. zero_image_recovery is checked against stock MATLAB's own particle framing; outoffocus re-derives the hollow diagnostics and must match the stored columns bit-for-bit before its correction is applied; overload_accounting_fix reproduces the port's sum_IntArr on unaffected seconds before touching the affected ones.
Where MATLAB never ran the code, no oracle can exist; the module says so and is validated structurally.
Still open
A 2DC diode-size discrepancy between two source files is open: real 2DC data is held (a 1995 ACE-1 file, probe C1, 25 um), but the comparison that would settle it has not been made. The per-particle classifier (ml_accept_reject.py) has a heuristic and a numpy baseline; the deep model is not built, because the limit is a labelled set of ~21 particles.