Background

Data in Water has varying levels of precision that is needed.  For example, if we're measuring the depth on a 20 ft deep tank, measuring down to the millimeter probably isn't needed, but down to a tenth of a foot is probably more appropriate.  Similarly, if we're measuring pH, usually one or two decimals (i.e. 7.1 or 6.97) is probably more appropriate than 4 decimals (i.e. 7.12935).  The reasons for this depend on the Detection Limits of instruments, the physical limitations of things like tank gauges and scales, as well as the requirements from regulators or downstream calculations that aggregate one or more value.


Additionally, best practices in math and science dictate that when we are doing math on values to calculate something else, we typically want to keep the highest level of precision that we have available through the calculation. This article explores some of the concepts involved in both deciding what level of precision is appropriate to store, as well as how you can change the level of precision for downstream calculations taking into account the various level of significant figures in your upstream calculations. We will unpack how we can display and report precision different than we store precision. 


Full Precision Data Handling Explained

Full Precision data handling means that a device (calculator or calculation engin), like Waterly, retains all the decimal points in your data "behind the scenes", even beyond what is displayed on-screen, similar to spreadsheets and calculators. This approach ensures that every calculation, analysis, or report generated is based on the most detailed and accurate data available.  When we take a very precise measurement and calculate something else from it using a conversion factor, it is typically not appropriate to “drop precision” (round) prior to the calculation.


Taking “Reasonable” Readings

Sometimes we see operators taking readings that are beyond the “natural precision” of an instrument.  For example, if you are trying to read the level of a 4 ft (or similar) diameter chemical tank like shown here, chances are, you can make the reading within maybe a half inch or so.  For a small, <1ft diameter tank, you would probably want quarter inches measured.  It also depends on how quiescent (settled) the tank is…tanks that have faster addition or withdrawals will sometimes have a liquid level that will “slosh” a bit and make very precise readings difficult.Or maybe you can take a measurement of a tenth of a foot. You cannot, reasonably, take a measurement down to a hundredth of a foot. Additionally, sometimes we see SCADA integrators and other instrumentation professionals who will configure a sensor to show you hundredths of a pound when you typically are measuring dozens or hundreds of pounds of usage each day. In cases like this, we encourage users to consider the reasonableness of the precision that is selected. Following is Waterly’s list of typical recommended levels of precision when it comes to measuring typical plant metrics in the water/wastewater industry:

  • Flow (plant flow meters >4”): MGal to 3 decimals (whole kGal) 
  • Flow (small diameter meters): Whole Gallons
  • Distances: Tenths of a Foot or Half Inches or Centimeters
  • Weights: Tenths of Pounds
  • Time: Whole Seconds

Waterly's Implementation of Precision

Displayed Data, Stored Data, Reported Data

To enhance readability, Waterly presents a rounded version of your data in data entry and on reports. This rounding is purely for display purposes, and all underlying calculations utilize the complete, unrounded Full Precision data.


Waterly Calculated Metric Example:

Consider this example scenario where Waterly is configured to display only 3 digits for a Metric that gets its data from a SCADA system.


Metric Value Input from SCADA: 1.4234

Waterly Displays: 1.423


When there is Calculation Metric that multiply Value by 2

Calculation: 1.4234 x 2 = 2.8468

Waterly Displays: 2.847


Here, Waterly uses the exact figure of 1.4234 for calculations but displays a rounded figure of 1.423 as configured, but when multiplied by 2 (to display an intermediate value) the actual result is 2.8468, but displayed as 2.847 as configured. Had we rounded the value without full precision before multiplying by 2, we would get an answer of 2.846. You may be saying that the difference is minuscule, but if you then are looking for the minimum, maximum, or average of these…say daily numbers, that error can propagate further. By maintaining full precision behind the scenes, a calculator (or spreadsheet, or Waterly) minimizes the downstream impact (inaccuracy) of rounding.  Here is a visual example in Waterly that shows the data above.



Click HERE learn how to adjust the number decimals to display in Waterly


When Precision can get Confusing in Watelry

When you have an INPUT set to one precision and a calculation that is set to another, data can look confusing.  See the example below where the Source Metric says 1.511 and the Max says 1.51 (that's pretty easy to understand), but now see #2, where it shows 2 because its precision is set to 0.  Then look at the "Above Value x 2" result, which one might expect would say 4, but it says 3.022.  That's because it is multiplying 1.511 (Source Metric) by 2, which results in 3.022 and NOT 4.



Summary

Accuracy: Your interim calculations are based on Full Precision data, ensuring the reliability of outcomes.

Clarity: By displaying rounded numbers, Waterly makes data easier to read and interpret.

Consistency: Full precision calculations help to assure consistent and reliable data handling across all of Waterly.


Similarly, when we calculate the daily/monthly/annual minimums, maximums, totals, and averages, we are using the Full Precision of each day's numbers, which means that we are as accurate as the inputted data.


Regulatory Realities

We have seen in some states (specifically, Indiana and Pennsylvania last we checked in summer 2024) where they have an Excel template that rounds numbers for the day PRIOR to monthly aggregation. We believe this practice to be wrong but may be a reality. If your state follows this practice in a template, we encourage you READ YOUR REGULATIONS, as we have also seen where the templates utilize incorrect rounding but the regulations do NOT specify that rounding intermediate calculations is required. In these cases, we recommend you contact your local inspector with a specific example for direction in writing and also encourage them to making note of the discrepancy in their templates. 


Fortunately, Waterly does support the scenario where we will enable our customers to round (truncate) intermediate values prior to downstream calculations, even though we don’t recommend it unless your state truly requires it. If your state specifically requires this, all you have to do is to create a support ticket, reference your state’s regulations, and we are happy to enable the functions to enable truncating numbers for you.


References and Related Articles