Precision

The precision of a number is the place value of the rightmost significant figure. For example, is precise to the hundredths place, and is precise to the tens place.

Precision-Based Rounding

In a previous module about decimals, we used precision-based rounding because we were rounding to a specified place value; for example, rounding to the nearest tenth. Let’s practice this with overbars and trailing zeros.

Remember, when the rounding digit of a whole number is a that gets rounded up to a , we must write an overbar above that .

Also, when the rounding digit of a decimal number is a that gets rounded up to a , we must include the in that decimal place.

Precision when Adding and Subtracting

Suppose the attendance at a large event is estimated at people, but then you see people leave. Is the new estimate ? No, because the original estimate was precise only to the nearest thousand. We can’t start with an imprecise number and finish with a more precise number. If we estimated that people had left, then we could revise our attendance estimate to because this estimate maintains the same level of precision as our original estimate.

Don’t round off the original numbers; do the necessary calculations first, then round the answer as your last step.

If you are multiplying by an exact number, you can consider this a repeated addition. For example, suppose you measure the weight of an object to be ounces and you want to know the weight of three of these objects; multiplying times is the same as adding ounces. The precision is still to the hundredths place. The issue of significant figures doesn’t apply to exact numbers, so it would be wrong to treat as having only one sig fig. (Treat exact numbers like royalty; their precision is perfect and it would be an insult to even question it.)

Greatest Possible Measurement Error (GPE)

Suppose you are weighing a dog with a scale that displays the weight rounded to the nearest pound. If the scale says Sir Barks-A-Lot weighs pounds, he could weigh anywhere from pounds to almost pounds. The true weight could be as much as pounds above or below the measured weight, which we could write as .

Now suppose you are weighing Sir Barks-A-Lot with a scale that displays the weight rounded to the nearest tenth of a pound. If the scale says Sir Barks-A-Lot weighs pounds, we now know that he could weigh anywhere from pounds to almost pounds. The true weight could be as much as pounds above or below the measured weight, which we could write as .

As we increase the level of precision in our measurement, we decrease the greatest possible measurement error or GPE. The GPE is always one half the precision; if the precision is to the nearest tenth, , the GPE is half of one tenth, or five hundredths, . The GPE will always be a in the place to the right of the place value of the number’s precision.

Another way to think about the GPE is that it gives the range of values that would round off to the number in question. Back to weighing Sir Barks-A-Lot: tells us a lower value and an upper value. is the lowest weight that would round up to . Similarly, is the highest weight that would round down to . Yes, perhaps we should say or , etc., for the upper limit here, but it is easier to just say and agree that is the upper limit even though it would round up instead of down. Using inequalities, we could represent as the range of values instead.

When you are asked to identify the GPE, it may help to think “What are the minimum and maximum numbers that would round to the given number?” For example, suppose the attendance at a Portland Thorns match is estimated to be people. This number is precise to the nearest thousand. The minimum number that would round up to would be (because would round down to ), and the maximum number that would round down to would be just below (because would round up to ). Because these numbers are each away from , the GPE is . If the estimate of is correct to the nearest thousand, we know that the actual attendance is within 500 of .

Recall from the previous module that the accuracy of a measurement is the number of significant figures. Let’s put together the ideas of accuracy, precision, and greatest possible measurement error.

Here is a summary of the important terms from these two modules. It is easy to get them mixed up, but remembering that “precision” and “place value” both start with “p” can be helpful.