A Framework for Quality Control Processing of Historical Turbidity Datasets

Historical turbidity records are valuable long after a monitoring campaign has ended. They can reveal sediment pulses, dredging impacts, storm-driven runoff, algal growth, resuspension, and gradual changes in catchment condition. Their usefulness depends on more than the number of observations. A dataset must retain a clear connection between each value, the instrument that produced it, the water it represented, and the processing applied before publication.

Turbidity is commonly reported in nephelometric turbidity units, including NTU and FNU, although the measurement response depends on optical geometry, wavelength, calibration material, particle size, colour, and fouling. Suspended-solids concentration is related to turbidity in many projects, but it is not interchangeable with it. A reliable quality-control process therefore preserves the original turbidity measurement instead of silently converting every record into a sediment estimate.

Older files often combine data from different sensors, loggers, deployments, laboratories, and contractors. Time zones may have changed, timestamps may have been rounded, and a monitor may have been replaced without a corresponding change in the file name. A defensible review treats these complications as evidence to be assessed rather than errors to be erased.

This approach is particularly important in Australia, where a dataset may span a dry season and “the wet”, a cyclone, a port expansion, and several monitoring contractors. Records from the Murray–Darling Basin, the Great Barrier Reef catchments, or a Western Australian mine-water program can be used for different regulatory and engineering purposes, but only when their limitations are visible.

Define The Intended Use And Data History

Quality control begins by stating what the dataset is meant to support. A record used to assess a dredging plume near Gladstone requires different scrutiny from a long-term stream turbidity series used for catchment modelling. Threshold exceedance assessment may depend on short peaks, while trend analysis may be more sensitive to changes in baseline, sampling frequency, and instrument configuration.

Write a short data-use statement before editing values. It should identify the monitoring locations, period covered, expected time resolution, reporting units, decision thresholds, and whether the output is intended for screening, compliance, research, or operational control. This statement prevents a legitimate high reading from being removed simply because it is inconvenient for a particular analysis.

Next, build a provenance register for every file and data segment. Record the source organisation, file name, sensor model, serial number, logger, deployment dates, location, depth, calibration history, firmware where available, laboratory method, and person or system that exported the data. Include the original file format and a checksum if the record is being managed under a formal data-governance system.

The register should also capture gaps and changes. A move from a D&A optical turbidity monitor to another manufacturer’s sensor, for example, may create a step change even when both instruments were operating correctly. Information about optical sensing principles and deployment context can be found in the technology overview, which helps reviewers connect recorded values with the behaviour of the measurement system.

Preserve Raw Values Before Applying Rules

The original export should be retained in read-only form. Do not overwrite raw readings with corrected values, interpolate missing periods in the source file, or replace instrument error codes with blanks without keeping the original representation. Create a working copy and assign each observation a quality flag, processing version, and reason code.

A practical flag set can distinguish valid, suspect, rejected, missing, below detection or instrument range, and not assessed. A binary pass/fail field is rarely sufficient. A value can be valid for a descriptive plot but suspect for a regulatory comparison because the sensor was fouled or the data interval was unusually long.

Keep separate fields for the recorded value, converted value, corrected value, and final value used in analysis. Store the conversion formula, constants, unit assumptions, and processing date alongside the output. This makes it possible to reproduce a report months later when a client, regulator, or project auditor asks why a peak was excluded.

Do not treat a blank as zero. Zero turbidity is a measurement result with a physical meaning, whereas a blank may indicate a communications failure, a logger pause, a removed sensor, or a value outside the configured range. The distinction matters during Australian storm events, when a short communications outage can coincide with the largest sediment pulse in the entire season.

Check Time, Units, And Measurement Continuity

Time errors are among the most damaging problems in historical environmental data. Check whether timestamps represent local time, UTC, daylight-saving time, logger time, or the time at which a file was downloaded. In Queensland, daylight saving is not used, while New South Wales and Victoria change clocks; a regional project spanning states can therefore contain an apparent one-hour shift that is administrative rather than environmental.

Compare the turbidity record with deployment logs, rainfall, water level, flow, pump status, and site photographs. A sudden shift at exactly midnight, at a daylight-saving transition, or after a logger reboot is more likely to be a time or configuration problem than a natural event. Convert all records to a documented standard while retaining the original timestamp and offset.

Units and scaling require the same discipline. Check whether the data logger stored integers with a scale factor, whether a decimal point was lost during export, and whether the instrument reported NTU, FNU, raw counts, percentage transmission, or a manufacturer-specific index. A calibration certificate may use a different unit convention from the monitoring database, so the conversion must be recorded rather than inferred from the column heading.

Sampling intervals can also change without warning. Resampling five-minute observations to hourly means, maxima, or medians can hide short plume peaks or give greater weight to periods with more observations. Define the aggregation rule, minimum valid count, treatment of gaps, and timestamp convention before generating daily or monthly summaries.

Identify Instrument And Deployment Effects

Optical turbidity sensors respond to suspended particles in the measurement volume, but the response is influenced by particle shape, colour, size distribution, bubbles, biofilm, and sediment composition. A reading from a clear-water river is not automatically comparable with a reading from a marine dredging plume. The relationship between turbidity and suspended solids should be established with site-specific paired samples where concentration estimates are required.

Review each deployment for fouling, burial, exposure, shading, movement, and orientation. Biofouling often produces a gradual rise or a slowly varying offset, while a loose mounting may generate repeated spikes linked to waves or vibration. Bubbles can cause sharp, short-lived peaks, particularly near aeration, boat traffic, culverts, spillways, or strong tidal flows.

Use maintenance records and field notes as evidence in the quality decision. A sensor cleaned on a known date may show a step down that is genuine instrument recovery rather than a sudden reduction in sediment. Conversely, a constant flat trace during a period of changing river stage may indicate a blocked optical window, frozen logger value, or failed communications path.

Instrument replacement deserves explicit treatment. Where two sensors overlap, compare them during stable and variable conditions, using robust statistics rather than a single ratio. Avoid forcing the older series onto the newer one unless the purpose, overlap length, calibration evidence, and uncertainty support that adjustment. Often the most honest solution is to retain separate segments and include the change point in the metadata.

Apply Evidence-Based Screening And Review

Automated tests provide consistency, but they should identify observations for review rather than make every final decision. Start with physical range checks based on the instrument specification and site conditions. Then apply rate-of-change, persistence, repeated-value, spike, gap, and sensor-status tests. A limit suitable for a small freshwater creek may be inappropriate for a turbid estuary or a dredging site.

Use contextual comparisons to reduce false rejection. A high turbidity value supported by rising stage, rainfall, upstream sensors, and field observations is likely to be an important event. A high value appearing in one channel while nearby sensors remain stable may deserve investigation, especially if the record also contains a battery alarm or a suspiciously short duration.

Visual review should occur at several scales: the full record, monthly or seasonal windows, individual events, and the minutes surrounding flagged points. Plot raw and processed data together, mark maintenance dates, and show missing intervals. Reviewers can then see whether an algorithm has removed genuine storm peaks, created artificial steps, or filled a gap across an instrument change.

For each rejected or adjusted observation, retain a reason code such as “out of range”, “fouling confirmed”, “timestamp unresolved”, “duplicate export”, or “bubble interference suspected”. If evidence is insufficient, use a suspect flag instead of rejection. Guidance, historical product information, and common questions about supported instruments may be available through the technical FAQ, but project-specific decisions should still be documented in the dataset itself.

Validate Results And Publish An Audit Trail

Validation should test both the cleaned record and the process that produced it. Calculate the proportion of valid, suspect, rejected, and missing observations by site and period. Compare these proportions before and after each major processing step. A dataset that appears smoother after quality control may simply have lost the variability that matters.

Assess event retention as well as general completeness. For each major rainfall, flood, dredging activity, or tidal cycle, confirm whether the processed record preserves the timing, peak, duration, and recovery pattern. Summaries should state whether maxima are observed values, interval means, interpolated estimates, or values affected by censoring.

A reproducible data package normally includes the raw files, cleaned data, flag dictionary, provenance register, calibration and maintenance records, processing code or transformation log, validation plots, and a release note. The release note should identify known limitations, unresolved periods, sensor changes, and any relationship used to estimate suspended solids from turbidity.

Terminology must be consistent across the package. Definitions for terms such as nephelometric turbidity, suspended solids, fouling, optical backscatter, detection limit, and quality flag can be checked against a relevant water-quality glossary. Clear definitions help consultants, port operators, researchers, Traditional Owner organisations, and regulators interpret the same dataset without assuming that similar words represent identical measurements.

A strong historical record does not pretend that every observation is equally certain. It shows what was measured, how the measurement was assessed, which evidence supports each decision, and where uncertainty remains. That transparency gives Australian monitoring programs a sound basis for comparing seasons, evaluating works, investigating sediment transport, and integrating older observations with new sensor deployments.