Background

Often, in water and wastewater treatment, operations staff is not onsite every day.  It is common (especially for smaller and mid-sized systems) to trust the automation and not personally check things every day.  However, often when we skip reading a meter or a scale weight manually, we'll estimate the usage or the data from the last reading as the difference between readings divided by the number of days we skipped.  Technically, this is called linear interpolation and it is a well-documented and accepted as a way to “construct new data points” in a set of data that might be missing. Waterly, along with other data management tools, supports the usage of interpolation as it is particularly beneficial for non-calculated, continuous metrics (readings) where daily data collection is impractical such as meter heads, chemical scales, and tank levels.  It lets operators and water quality supervisors to have a more complete picture of their data in situations where linear interpolation is appropriate.  This article explores how it works, as well as its appropriateness in the water/wastewater industry.  Interpolation should never be used to represent water quality parameters that are normally a result of a laboratory test.  If you are ever unsure about if interpolation is appropriate for data, you should consult with your regulators.

Interpolation should never be used to represent water quality parameters that are normally a result of a laboratory test.

Summary

Data Interpolation, when implemented in an appropriate context, can be a useful tool in efficient water data management, tailored to support metrics that rise and fall in expected patterns over days.  

Details

• Interpolation is usually Suitable for 'Continuous' Metrics: Suitable for metrics with steady consistent pattern, such as gradual and predictable increases or decreases.  This typically includes metrics like chlorine tank level drawdowns, scale weights, or blower runtime hours, where values can be reasonably inferred even when not recorded daily

• Inappropriate for Discontinuous Metrics: Interpolation is not suitable for measurements like rainfall, laboratory or instrument test results, or readings from a chlorine analyzer where the data doesn’t follow an expected trend.

Examples 

Tank Level

We take the two known readings of 130 and 100, find the difference between the three (3) days, and divide the difference by the number of days, or 30 / 3 = 10.  We then apply that back in the series to fill in the data points.  Here's what it would look like visually, with the two interpolated numbers in gray:

Turning Interpolation on for Metrics

Only Supervisors and above can enable Interpolation in Waterly.

You must be in Edit Mode to enable interpolation. In edit mode:

1. Click the Edit Pencil next to the Metric you want to Enable for Interpolation.
2. Check Interpolate Data in the Metric Editor 
3. Click Save

Here's what it would look like visually:

Data Entry and Reporting

Data Entry & Visualization

Waterly tries to communicate differences in data using as little clutter as possible, while being responsible.  As a result, there are icons to show what metrics are enabled for interpolation, and what data points are interpolated.

• Enabled for Interpolation: Interpolated values are marked clearly with an icon on the metric input box with an all gray line with an empty circle in the middle, like this:
  

• Here is what the icon looks like on the screen when the metric is enabled for interpolation:

• Interpolated Data Point: When a datapoint is generated from interpolation, the icon changes to have a hollow center indicating it is an interpolated value between two real values:

• PLUS, if you hover or tap on the icon, it will tell you that data point is an interpolated data point. You can always write over an interpolated data point with a (real) data point. New data points will be recalculated. Here's what a data point looks like that is interpolated:

Reporting

All data points that are interpolated will be marked with a footnote. Additionally, any calculated values that are based on interpolated data will also be marked with an additional footnote.

Here's what the report will look like in a report with some notes:

Limitations

Interpolation is not unlimited.  For longer periods of time, you should ideally take a (real) reading.  Remember to check with your regulator or your organization, which may have more restrictive rules surrounding interpolation.  Following are the specific limitations Waterly has when interpolating data:

• 35-Day Interpolation Max: Interpolation is limited to gaps of up to 35 days, and is more ideal for smaller gaps, like a weekend or weekly or monthly data capture.  It will not work for larger than 35 days (no data will backfill).

• Daily Metrics: Interpolation is only available on Daily metrics, not on Interval metrics (typically used for shift data capture).  

• Linear Interpolation: Interpolation only creates a linear interpolation across the gap.  (Difference / # of days)

• Continuous Data Requirement: Interpolation should ONLY BE USED for metrics with an expected, or continuous trend.  It is unsuitable for erratic or unpredictable data types.  If you are not sure, contact a licensed engineer or your organizational leadership for direction.

• Calculated Metrics: Interpolation is not appropriate and cannot be enabled on a calculated metric, as there could be conflicting values.

• Limited Units: We ONLY enable the following units to be interpolated at this time. If you think we should allow more than this list, please let us know with a support ticket and we will consider adding:

Technical Notes

Following are some technical notes about the behavior of the app when you go back and remove/change data points that were interpolated. If you've made it this far, congratulations...we may have a job for you.  We care a LOT about data, precision, and accuracy, so this will explain how we handle some edge cases with interpolation.

Deleting Values When the Metric Is Interpolated

In the following area, we will use these symbols to represent datapoint types:

O - Interpolated Data Point

X - Manually entered Data Point

_ - Blank / Null Data Point

X O O X O O X

When deleting the center bold data point the values will then attempt to re-interpolate across that span and form - X O O O O O X - unless the span between the points is greater than 35 days. In that case it would form X _ …. _ _ _ _ X

X O O O O X

When deleting the last manually entered data point with no others in that direction within 35. It will delete the interpolated data: X _ _ _ _ _

X O O O O O X

Deleting an interpolated data point will remove that datapoint and not re-interpolate data: X O O O _ O X