Software for measurement uncertainty evaluation
The 'Guide to the expression of uncertainty in measurement' (GUM) [2] is the primary document regarding the evaluation and reporting of uncertainty in measurement. Working Group 1, 'Expression of uncertainty in measurement', of the Joint Committee for Guides in Metrology (JCGM) is undertaking work to promote and extend the application of the GUM through the preparation of supporting documents [1]. The first such document, GUM Supplement 1 [3], is concerned with the use of a Monte Carlo method as an implementation of the propagation of distributions for uncertainty evaluation.
GUM Supplement 1 includes four example problems, concerning an additive measurement model, mass calibration, comparison loss in microwave power meter calibration, and gauge block calibration. NPL provides NPLUnc_101 [4], software to support the use of the GUM and GUM Supplement 1 by enabling users to apply the approaches to uncertainty evaluation described in these documents to those example problems. NPLUnc_101 is intended to allow users to reproduce the results presented in tables and figures contained within GUM Supplement 1. The software is also intended to help users learn about the methods of uncertainty evaluation described in the GUM and GUM Supplement 1 by enabling them to experiment with (a) different information about the input quantities in the models defining the example problems, e.g., estimates and associated standard uncertainties for the input quantities in those models, and (b) different values for the parameters controlling the application of those methods, e.g., the number of trials in an application of a Monte Carlo method. The software does not allow users to undertake uncertainty calculations for measurement models other than those for the example problems given in GUM Supplement 1. However, NPL has published specifications [5] of the algorithms implemented in the software.
The software NPLUnc_101_source presented here is intended to promote further the methods for uncertainty evaluation described in the GUM and GUM Supplement 1. Source code, written in the MATLAB programming language [6], is provided for applying the GUM uncertainty framework and a Monte Carlo method to a measurement model of the form
with a scalar output quantity and a general number of uncorrelated input quantities. Each method delivers an estimate of the output quantity in the measurement model, the standard uncertainty associated with the estimate, a coverage interval for the output quantity corresponding to a specified coverage probability, and an approximation to the probability density function for the output quantity.
The Monte Carlo procedure may be applied either non-adaptively or adaptively. In a non-adaptive application of the procedure, a fixed number of Monte Carlo trials, specified by the user, is undertaken and a test of whether the results obtained have stabilized in a statistical sense is performed at the end of the calculation. The test is based on a numerical tolerance calculated in terms of a number of significant decimal digits regarded as meaningful in the value of the standard uncertainty. In an adaptive application of the procedure, the calculation is terminated when either (a) the results obtained have stabilized, or (b) a maximum number of trials, specified by the user, has been undertaken.
In the case that the results obtained from the Monte Carlo procedure have stabilized, those results are used to validate the results obtained from the GUM uncertainty framework. The basis of the validation is also a numerical tolerance calculated in terms of a number of significant decimal digits regarded as meaningful in the value of the standard uncertainty. A sufficient number of Monte Carlo trials should be undertaken in obtaining results from a Monte Carlo method for the purpose of validating those from the GUM uncertainty framework, and it is recommended that the numerical tolerance for testing for stabilization of the Monte Carlo results is chosen to be no bigger than one fifth of that used for validation of the GUM uncertainty framework results.
Information on how a numerical tolerance is evaluated in terms of a number of significant decimal digits is given in GUM Supplement 1 [3] and the user manual [4, section 3.1.5] for NPLUnc_101. In the latter document, guidance is provided on how to set the number of significant decimal digits to give a particular numerical tolerance, which may require the use of a number of significant decimal digits that is not an integer.
Users of MATLAB may run the software NPLUnc_101_source to obtain the results included in this document for a particular problem of uncertainty evaluation. They may also modify the definition of the problem, including the measurement model, and apply the methods for uncertainty evaluation to the modified problem. The source code is provided with extensive commenting and links to the specifications [5] to help practitioners develop their own implementations of the software, e.g., in other programming languages. Some use of MATLAB intrinsic functions is made, such as for linear interpolation, and implementations of such functions would also be needed.
The software NPLUnc_101_source has a user interface that is much simpler than that for NPLUnc_101 [4]. Additionally, very few internal checks are undertaken, and consequently the software cannot be expected to be as robust as NPLUnc_101. The reasons for simplifying the software in these ways are to help with the readability of the code and to help make transparent the algorithms implemented and their relationship to the specifications [5]. Users should therefore exercise care when running the software on a new or modified problem of uncertainty evaluation.
The software is organized as follows. This file is NPLUnc_101_source.html. The files GUM_calculation.html and MCM_calculation.html contain functions implementing, respectively, the GUM uncertainty framework and a Monte Carlo method for uncertainty evaluation. The file GUM_validation.html contains functions for displaying and comparing the results obtained from the two methods. Finally, the file model_additive.html contains an example of a function implementing a measurement model. The example considered is the additive model
MATLAB users can use the MATLAB 'grabcode' function to retrieve the M-code from these HTML files, and run the code directly. Information about how the uncertainty evaluation problem to be solved is defined (and can therefore be modified) is given in the sections below.
The software is provided with a Software licence agreement (REF: MSC/L/08/007) and the use of the software is subject to the terms laid out in that agreement. By retrieving and running the M-code, the user accepts the terms of the agreement.
The software should be used in conjuction with the GUM [2], GUM Supplement 1 [3], the user manual [4], and the specifications [5] of software implementing the GUM uncertainty framework and a Monte Carlo method.
Contents
- Defining the measurement model
- Defining the probability distributions assigned to the input quantities
- Defining the coverage probability and type of coverage interval
- Defining the controls for the Monte Carlo calculation
- Defining the controls for displaying and validating the results
- Calculation based on the GUM uncertainty framework
- Calculation based on a Monte Carlo method
- Display and validation of the results
- References
- Acknowledgements
- Program identifier
Defining the measurement model
model is a string containing the name of a function for evaluating the measurement model for values of the input quantities and (optionally) evaluating sensitivity coefficients. In this example [3, section 9.2], the model is a simple additive model defined by the function model_additive.
model = 'model_additive';
Defining the probability distributions assigned to the input quantities
Each input quantity in the measurement model is assigned a probability distribution, described by a probability density function, which may be (1) a Gaussian (or normal) distribution, (2) a scaled and shifted t-distribution, (3) a rectangular (or uniform) distribution, (4) a curvilinear trapezoidal distribution, or (5) a U-shaped distribution. (This list includes all the univariate probability distributions encountered in the examples of GUM Supplement 1. There is no consideration of joint probability distributions, such as the multivariate Gaussian distribution that is used in the example of comparison loss in microwave power meter calibration.)
pdfin is a matrix of N rows and 4 columns, the ith row of which defines the probability distribution for the ith input quantity. For a measurement model with a single input quantity, pdfin can take the following forms:
pdfin = [1, mu, sigma, inf] defines a Gaussian distribution with expectation mu and standard deviation sigma (the fourth element of pdfin is not used);
pdfin = [2, mu, sigma, nu] defines a t-distribution with shift parameter mu, scale parameter sigma and degrees of freedom nu;
pdfin = [3, a, b, inf] defines a rectangular distribution with lower limit a and upper limit b (the fourth element of pdfin is not used);
pdfin = [4, a, b, r] defines a curvilinear trapezoidal distribution with lower limit a, upper limit b and (fractional) reliability r for the semi-width;
pdfin = [5, a, b, inf] defines a U-shaped distribution with lower limit a and upper limit b (the fourth element of pdfin is not used).
Guidance on the assignment, in some common circumstances, of probability density functions for the input quantities in a measurement model is given in GUM Supplement 1 [3, section 6]. Information about the specifications of the probability distributions listed above is given in the user manual [4, section 3.1.1] for NPLUnc_101.
In this example [3, section 9.2.2], there are four input quantities, each assigned a Gaussian distribution, and each with an estimate (expectation) of zero and an associated standard uncertainty (standard deviation) of unity.
pdfin = [ 1, 0, 1, inf 1, 0, 1, inf 1, 0, 1, inf 1, 0, 1, inf ];
Defining the coverage probability and type of coverage interval
The required coverage interval is defined by the coverage probability p, which can be 0.90, 0.95 or 0.99, and type, which can be 'Shortest' or 'Symmetric'. The shortest coverage interval is the coverage interval with the shortest length among all coverage intervals having the same coverage probability. The (probabilistically) symmetric coverage interval for a quantity is such that the probability that the quantity is less than the smallest value in the interval is equal to the probability that the quantity is greater than the largest value in the interval. In this example, the probabilistically symmetric 95 % coverage interval is to be determined.
p = 0.95;
type = 'Symmetric';
Defining the controls for the Monte Carlo calculation
controls is a row vector of five elements and contains controls for the Monte Carlo calculation as follows:
controls(1) determines whether the calculation is adaptive (1) or not (0);
controls(2) is the maximum number of Monte Carlo trials to be undertaken (as a multiple of 10 000);
controls(3) is the number of histogram bins for defining an approximation to the probability density function for the output quantity;
controls(4) is the initial state of the random number generator used in generating random draws from the probability distributions for the input quantities;
controls(5) is the number of significant decimal digits regarded as meaningful in the value of the standard uncertainty when testing for stabilization of the results.
In this example, the calculation is to be adaptive, but no more than 1 000 sets of 10 000 trials are to be undertaken, the approximation to the probability density function for the output quantity is to be based on 1 000 uniformly spaced histogram bins, and the initial state of the random number generator is set using the internal clock. When testing for stabilization of the results, a value of 2.7 is used for the number of significant decimal digits regarded as meaningful in the value of the standard uncertainty, which corresponds to a numerical tolerance of 0.01 [4, section 3.1.5].
controls = [1, 1000, 1000, sum(1000*clock), 2.7];
Defining the controls for displaying and validating the results
Nbins is the number of histogram bins used to display an approximation to the probability density function for the output quantity returned by the Monte Carlo calculation. nval is the number of significant decimal digits regarded as meaningful in the value of the standard uncertainty when validating the results obtained from the GUM uncertainty framework against those obtained from a Monte Carlo method.
In this example, 50 histogram bins are used to display an approximation to the probability density function. Validation is based on regarding 2 significant decimal digits as meaningful in the value of the standard uncertainty, which corresponds to a numerical tolerance of 0.05 (five times that used for testing for stabilization of the Monte Carlo results, above) [4, section 3.1.5].
Nbins = 50; nval = 2;
Calculation based on the GUM uncertainty framework
See GUM_calculation.
[yGUM, uyGUM, IyGUM, pdfGUM] = GUM_calculation(model, pdfin, p);
Calculation based on a Monte Carlo method
See MCM_calculation.
[yMCM, uyMCM, IyMCM, pdfMCM, conv] = ...
MCM_calculation(model, pdfin, p, type, controls);
Display and validation of the results
See GUM_validation.
GUM_validation(yGUM, uyGUM, IyGUM, pdfGUM, yMCM, uyMCM, IyMCM, pdfMCM, ...
conv, Nbins, nval);
Results from GUM uncertainty framework: y = 0 u(y) = 2 I(y) = -3.92, 3.92 Results from a Monte Carlo method: y = -0.0032138 u(y) = 1.99621 I(y) = -3.91441, 3.91303 MCM calculation has stabilized GUF is validated against MCM

References
[1] W Bich, M G Cox and P M Harris, Evolution of the 'Guide to the expression of uncertainty in measurement' (Metrologia 43, S161-7, 2006)
[2] BIPM, IEC, IFCC, ISO, IUPAC, IUPAP and OIML, Guide to the expression of uncertainty in measurement (ISBN 92-67-10188-9, corrected and reprinted, 1995)
[3] BIPM, IEC, IFCC, ILAC, ISO, IUPAC, IUPAP and OIML, Evaluation of measurement data - Supplement 1 to the 'Guide to the expression of uncertainty in measurement' - Propagation of distributions using a Monte Carlo method (JCGM 101, Joint Committee for Guides in Metrology, 2008)
[4] M G Cox, P M Harris and I M Smith, Software for GUM Supplement 1: User Manual, 2011
[5] M G Cox, P M Harris and I M Smith, Software specifications for uncertainty evaluation, 2010
[6] MATLAB (Copyright The MathWorks, Inc.) www.mathworks.com
Acknowledgements
The work described here was supported by the National Measurement System Directorate of the UK Department for Innovation, Universities and Skills as part of the NMS Software Support for Metrology programme.
Program identifier
NPLUnc_101 Open-Source Software (Release 1.1): Software for measurement uncertainty evaluation
M G Cox, P M Harris and I M Smith, Mathematics and Scientific Computing Group, National Physical Laboratory, UK
Crown copyright 2011