6.5. VADER: Trending Analysis for Code/Data Validation

S. Hart, J. Clarity, L. Fassino

VADER (Validation Analysis Data Evaluation Resource) is a new module in SCALE 6.3 derived from the legacy USLSTATS program. VADER is a tool that allows the determination of bias and bias uncertainty for criticality safety computational methods. The older USLSTATS program, written in Java, existed outside of SCALE and provided tools to calculate only the confidence band with administrative margin (sometimes called USL-1) and the single-sided uniform width closed interval (USL-2). For normality testing it offered only a crude chi-squared normality test that had no user-configurable options and presented a simple pass/no-pass functionality.

VADER has been written completely in C++ using modern coding practices and standards. It has been moved to be part of the SCALE code package and is run through the normal SCALE executables. To support the VADER methods, large updates were made to the statistical capabilities of SCALE. Interfaces to known statistical distributions were added and many of the statistical calculations required for VADER were made available to all SCALE sequences. In addition to being able to process legacy USLSTATS inputs, VADER adds support for the single-sided tolerance band method (from NUREG/CR-6698 [VADER-DRWT01]) as well as two non-trending methods: the “historical” nonparametric and parametric methods, also as described in NUREG/CR-6698 [VADER-DRWT01]. To support determination of the appropriate method for a given dataset, VADER also includes the t-test for trend significance and the Anderson-Darling and chi-squared tests for normality.

VADER has also been integrated into the Fulcrum GUI used by SCALE. This allows VADER to benefit from the auto-complete, templating, and input validation features offered by Fulcrum. In addition, Fulcrum can plot the results given by VADER, and the plot files have been expanded to include the new methods available in VADER.

The methods outlined herein are consistent with the descriptions available in “Determination of Bias and Bias Uncertainty for Criticality Safety Computational Methods” [VADER-CMM+25], which provides complete detail on the origins and limitations for these methods. Some guidance on the usage of these methods and tests is provided below, but the user is advised to reference [VADER-CMM+25] for more information.

6.5.1. Input

The VADER input is based on the SON input syntax common in other modernized SCALE codes such as ORIGEN and ORIGAMI. The SON input format consists of key/value pairs where the value can be either a single item, an array of items, or other SON blocks.

The VADER input generally has one level of depth. There are no nested SON blocks. Input data is primarily structured using the SCALE array format. An overview of top level items is provided in Table 6.5.1, which are elaborated upon in the detailed input section.

Table 6.5.1 Summary of top level options for VADER
Keyword	Required	Description
data	Yes	Data entered in triplets in the form `X Y unc.`.
title	No	Title to give the case.
parameters	No	Global parameters that apply to each test or trend.
trend_values	No	User-specified trend values to evaluate in selected method(s).
methods	Yes	Selected methods to apply to the data.
tests	No	Statistical tests to apply to the data.

6.5.1.1. Legacy Input

For backwards compatibility VADER supports reading the original USLSTATS format. This format is restricted to USL methods USL1 and USL2. Results should match those obtained using the Java-based USLSTATS program with some exceptions detailed below.

The extrapolation parameter has been updated to be more rigorous. Only USL1 can do extrapolation and, if turned off, values outside of the range will be labeled in the table as N/A. USL2 will always give N/A for values outside of the range.
The USL1 method one will always use the maximum band width inside the x-range, \(W\), inside of the range and then the larger of \(W\) or \(w(x)\) outside of the range (if extrapolating). This differs from legacy USLSTATS that used w(x) across the entire range if extrapolating.
The administrative margin parameter in legacy USLSTATS uses a default value of 0.03, which differs from the VADER default value of 0.05.

6.5.2. Detailed Input

6.5.2.1. `data`

The data input array is a 1-dimensional array consisting of data triplets. There is no punctuation between the triplets. The data is entered in the order \(X_1 \, Y_1 \, \sigma_1 \; X_2 \, Y_2 \, \sigma_2 \; ... \; X_N \, Y_N \, \sigma_N\).

Where

X	The trended value, usually something like enrichment or moderator density.
Y	The y value, usually something like C/E or eigenvalue.
\(\sigma\)	The standard deviation of Y.

VADER will emit an error if the number of entries in this array is not divisible by three.

Example 6.5.1 Snippet of data input

data=[2.4 0.9976 0.00122 2.6 0.9978 0.00022
      ...
      1.9 1.0011 0.00213 1.99 1.013 0.00012 ]

6.5.2.2. `title`

The title entry allows the user to specify a title for the case. The title is cosmetic and doesn’t affect any of the results. The title is echoed to the output in several places and is included on some of the graphs generated by VADER.

Example 6.5.2 title

title="My Very First VADER Case"

6.5.2.3. `parameters`

The parameters section can exist outside of any test or method. If a parameter is listed in this global section, then it will apply to any method or test that has that parameter available instead of the default value. Values in this global parameters section can be overridden in the specific method or test block for which the user wishes to override the global parameter. For a list of all parameters, their default values, and which tests and methods they are applied to, see Sect. 6.5.5.

Example 6.5.3 parameters array

parameters {extrapolate=yes admin_margin=0.03 }

6.5.2.4. `trend_values`

This array allows the user to insert values at which they wish the trend Y (typically the upper subcritical limit) values to be calculated. Extrapolation outside of the data range is only available if the method supports it and extrapolation is enabled.

Example 6.5.4 trend_values array

trend_values=[-0.1 0.2 2.4 10.0 ]

6.5.2.5. `methods`

This is where the user specifies the methods to enable and, optionally, gives method-specific parameters. Each method is followed by an array of the parameters the user wishes to set. Any omitted parameters are initialized with the default value (see Table 6.5.3). If the user wishes to enable a method with all default values, then an empty array can be given.

Example 6.5.5 methods block with some default methods and some specific parameters

methods {
  USL1 {} % uses default parameters
  CR6698 {weighted=yes admin_margin=0.03 chi_p=0.97 }
  parametric {confidence=0.9 }
}

6.5.2.6. `tests`

This block is similar to the methods block, but it allows the user to specify which tests they wish to enable. These tests are run on the data and generally either pass or fail and give a few statistical values as their result. Most of the tests implemented for VADER for now are normality tests.

Example 6.5.6 tests block

tests {
  chisq {} % default parameters
  t-test {confidence=0.98 }
}

6.5.3. Available Methods

The sections below indicate the input name to be used to activate the available method. The origins and use cases for these methods are provided in [VADER-CMM+25], which should be referenced to assist in determining which method is suitable for a given dataset.

The implementation of these methods is described with respect to calculating the bias, bias uncertainty, and the upper subcritical limit (USL) based on a dataset consisting of some independent variable, \(x_i\), with its associated \(k_{norm,i}\) and uncertainty, \(\sigma_i\).

The normalized \(k_{eff}\) (\(k_{norm}\)) is the ratio of an experiment’s calculated \(k_{eff}\) (\(k_{calc}\)) to its expected \(k_{eff}\) (\(k_{exp}\)), as shown in Eq. (6.5.1). Its respective combined uncertainty, \(\sigma_i\), is the squared sum of the associated relative variances for \(k_{calc}\) and \(k_{exp}\), scaled to convert to absolute uncertainty by \(k_{norm}\), as shown in Eq. (6.5.2).

(6.5.1)\[k_{norm} = \frac{k_{calc}}{k_{exp}}\]

(6.5.2)\[\sigma = k_{norm} \sqrt{\left(\frac{\sigma_{exp}}{k_{exp}}\right)^{2} + \left(\frac{\sigma_{calc}}{k_{calc}}\right)^{2}}\]

Typically, \(\sigma_{calc}\) is the stochastic uncertainty associated with a Monte Carlo calculation of \(k_{eff}\), and it may be referred to as \(\sigma_{MC}\) instead. Additionally, the combined uncertainty expression in Eq. (6.5.2) may be reduced to the square root of the sum of variances \(\sigma_{calc}^2\) and \(\sigma_{exp}^2\) as an approximation, as the difference between the absolute and relative uncertainties are typically very small given that the values for \(k\) in this context are expected to be near 1. This approximation yields the more commonly seen form in Eq. (6.5.3).

(6.5.3)\[\sigma = \sqrt{\sigma_{exp}^{2} + \sigma_{calc}^{2}}\]

Each method calculates the USL according to Eq. (6.5.4), where the bias, \(\beta\), is always less than or equal to zero; positive biases are not credited in the USL calculation for any method.

(6.5.4)\[USL = 1 + \beta - \sigma_{\beta} - M_A - M_D \text{ for } \beta \leq 0\]

In the above calculation, \(\sigma_{\beta}\) is the bias uncertainty, \(M_A\) is the administrative margin, and \(M_D\) is the margin for other deficiencies.

Note

The bias is not explicitly reported in the output; the adjusted \(k(x)\) value, Adj. k(x), is reported instead. The Adj. k(x) value is equivalent to \(1+\beta(x)\), and it is never greater than 1 since the bias must always be less than or equal to zero.

Additionally, the nonparametric margin (NPM) in the nonparametric method is the only instance where \(M_D\) exists and may be greater than zero, but it is not reported separately. Instead, the reported bias uncertainty incorporates the NPM when it is greater than zero, which is noted in the output when true.

If the \(k_{norm}\) data has a significant linear trend with respect to the independent parameter, then trending methods CR6698, USL1, and USL2 may be used. If there is no significant linear trend in the dataset but the data has a normal distribution, then the non-trending parametric method may be used. Otherwise, the nonparametric method, a non-trending, distribution-free method, may be used. See Available Tests for information on testing for trends and the normal distribution.

6.5.3.1. `CR6698` – Single-Sided Lower Tolerance Band

The CR6698 method uses the single-sided lower tolerance band method to calculate the USL. In this method, the linear fit of the bias is defined by Eq. (6.5.5), with a bias uncertainty defined by Eq. (6.5.6).

(6.5.5)\[\beta(x) = k_{fit}(x)-1\]

(6.5.6)\[\sigma_{\beta}(x) = S_p \left( \sqrt{2 F_{\alpha}^{(2,N-2)} \left[ \frac{1}{N} + \frac{(x-\bar{x})^2}{S_{xx}} \right] } + z_p \sqrt{\frac{N-2}{\chi^{2}_{1-\frac{\alpha}{2},N-2}}} \right)\]

The linear fit for \(k_{norm}\) as a function of \(x\) is defined by Eq. (6.5.7), where \(\beta_0\) and \(\beta_1\) are the intercept and slope of the linear fit defined in Eq. (6.5.8) and Eq. (6.5.9), respectively. The calculation of bias uncertainty relies on several statistical parameters as well, which are detailed below.

(6.5.7)\[k_{fit}(x) = \beta_0 + \beta_1 x\]

(6.5.8)\[\beta_0 = \bar{k}_{norm} - \beta_1 \bar{x}\]

(6.5.9)\[\beta_1 = \frac{S_{kx}}{S_{xx}}\]

Eq. (6.5.8) requires calculation of the average \(k_{norm}\) and \(x\) values, which may be calculated using either uncertainty weighted or unweighted methods. The unweighted expression for \(\bar{k}_{norm}\) is given by Eq. (6.5.10), and the weighted calculation is given by Eq. (6.5.11).

(6.5.10)\[ \bar{k}_{norm} = \frac{\sum_{i=1}^{N} k_{norm_i}}{N}\]

(6.5.11)\[ \bar{k}_{norm} = \frac{\sum_{i=1}^{N} \frac{k_{norm_i}}{\sigma_i^2}}{\sum_{i=1}^{N} \frac{1}{\sigma_i^2}}\]

In this equation, \(N\) is the number of points in the dataset, and \(k_{norm_i}\) and \(\sigma_i\) are the normalized \(k_{eff}\) and the associated combined uncertainty, given by Eq. (6.5.1) and Eq. (6.5.2), respectively. Likewise, the unweighted calculation for \(\bar{x}\) is given by Eq. (6.5.12), and the weighted calculation is given by Eq. (6.5.13).

(6.5.12)\[ \bar{x} = \frac{\sum_{i=1}^{N} x_{i}}{N}\]

(6.5.13)\[ \bar{x} = \frac{\sum_{i=1}^{N} \frac{x_{i}}{\sigma_i^2}}{\sum_{i=1}^{N} \frac{1}{\sigma_i^2}}\]

Note

Methods with both unweighted and weighted variations use either the unweighted or weighted expressions consistently throughout the calculation.

In Eq. (6.5.9) as well as in Eq. (6.5.6), \(S_{xx}\), the sum of squared deviations about the mean for the independent variable, is calculated using the unweighted method with Eq. (6.5.14) and for the weighted method with Eq. (6.5.15).

(6.5.14)\[S_{xx} = \sum^{N}_{i=1} (x_i - \bar{x})^{2}\]

(6.5.15)\[S_{xx} = \frac{ \sum^{N}_{i=1} \left( \frac{1}{\sigma_{i}^2}(x_{i} - \bar{x})^{2} \right)} { \frac{1}{N} \sum^{N}_{i=1} \frac{1}{\sigma_{i}^{2}}}\]

Similarly, the \(S_{kx}\) calculation is the product sum of deviations of \(x\) and \(k_{norm}\) about their respective means, given by Eq. (6.5.16) for the unweighted method and Eq. (6.5.17) for the weighted method.

(6.5.16)\[S_{kx} = \sum^{N}_{i=1} (k_{norm_i} - \bar{k}_{norm}) (x_{i} - \bar{x})\]

(6.5.17)\[S_{kx} = \frac{ \sum^{N}_{i=1} \left( \frac{1}{\sigma^{2}_{i}} (x_{i} - \bar{x}) (k_{norm_i}-\bar{k}_{norm}) \right)} { \frac{1}{N} \sum^{N}_{i=1} \frac{1}{\sigma^{2}_{i}} }\]

The calculation of the bias uncertainty per Eq. (6.5.6) requires the calculation of several other parameters: \(S_p\), the square root of the pooled variance; \(F_{\alpha}^{(2,N-2)}\), the inverse of the F probability distribution corresponding to a degree of fit 2 and \(N-2\) degrees of freedom for a significance of \(\alpha\); \(z_p\), the inverse of the standard normal cumulative distribution containing fraction \(p\) of the distribution; and \(\chi^{2}_{1-\frac{\alpha}{2},N-2}\), the inverse of the right-tailed probability of the chi-squared distribution.

Eq. (6.5.18) describes the calculation for \(S_p\) of the linear fit. It combines the variance of the fit, \(\sigma_{fit}^2\), and the average total variance, \(\bar{\sigma}^2\), which is sometimes referred to as the “within variance.” The average total variance is given by Eq. (6.5.19) for the unweighted method and by Eq. (6.5.20) for the weighted method. The unweighted calculation of \(\sigma_{fit}^2\) is given by Eq. (6.5.21) for the unweighted method and by Eq. (6.5.22) for the weighted method.

(6.5.18)\[S_p = \sqrt{\sigma_{fit}^2 + \bar{\sigma}^2}\]

(6.5.19)\[\bar{\sigma}^2 = \frac{\sum_{i=1}^{N}\sigma_{i}^{2}}{N}\]

(6.5.20)\[\bar{\sigma}^2 = \frac{N}{\sum_{i=1}^{N}\frac{1}{\sigma_{i}^{2}}}\]

(6.5.21)\[\sigma_{fit} = \sqrt{ \frac{\sum^{N}_{i=1} e^{2}_{i}} {N-2}}\]

(6.5.22)\[\sigma_{fit} = \sqrt{ \frac{ \frac{1}{N-2} \sum^{N}_{i=1} \frac{1}{\sigma_{i}^{2}} e_{i}^{2} } { \frac{1}{N} \sum^{N}_{i=1} \frac{1}{\sigma_{i}^{2}} }}\]

The calculation of \(\sigma_{fit}^2\) requires calculating the residual error, \(e_{i}\), for each datapoint with respect to the fit, as defined by Eq. (6.5.23).

(6.5.23)\[e_{i} = k_{norm_i} - k_{fit}(x_{i})\]

Note

The calculation of the linear fit–related parameters are necessary for the other trending methods, USL1 and USL2, as well. These methods only use the unweighted expressions.

6.5.3.2. `USL1` – Confidence Band with Administrative Margin

The USL1 method uses the confidence band with administrative margin method to calculate the USL. The bias is calculated according to Eq. (6.5.5) using the unweighted method described in CR6698 – Single-Sided Lower Tolerance Band, and the bias uncertainty is calculated using Eq. (6.5.24). The calculated bias uncertainty for a given x-value depends on whether it is inside or outside the input xrange. For values inside xrange, the bias uncertainty evaluated at \(x_{min}\) and \(x_{max}\), and the maximum is returned. This is represented by Eq. (6.5.25). Outside of xrange, \(\sigma_{\beta}(x)\) is returned, unless either \(\sigma_{\beta}(x_{min})\) or \(\sigma_{\beta}(x_{max})\) is higher, in which case the larger value will be returned.

(6.5.24)\[\sigma_{\beta}(x) = t_{1-\alpha,N-2} S_p \sqrt{1 + \frac{1}{N} + \frac{(x-\bar{x})^2}{S_{xx}}}\]

In this equation, \(S_p\), \(\bar{x}\), and \(S_{xx}\) correspond to unweighted calculations in Eq. (6.5.18), Eq. (6.5.12), and Eq. (6.5.14), respectively. The value \(t_{1-\alpha,N-2}\) is the inverse t-distribution corresponding to the desired confidence with \(N-2\) degrees of freedom.

(6.5.25)\[\sigma_{\beta} = \max(\sigma_{\beta}(x_{min}),\sigma_{\beta}(x_{max}))\]

6.5.3.3. `USL2` – Single-Sided Uniform Width Closed Interval Approach

The USL2 method uses the single-sided, uniform-width, closed-interval approach to calculate the USL. The bias is calculated according to Eq. (6.5.5) using the unweighted method described in CR6698 – Single-Sided Lower Tolerance Band, and the bias uncertainty for this method is calculated using Eq. (6.5.26).

(6.5.26)\[\sigma_{\beta} = C_{\alpha/p} \, S_p\]

Here, the square root of the pooled variance, \(S_p\), is as calculated per Eq. (6.5.18) using unweighted methods, and the parameter \(C_{\alpha/p}\) is defined in Eq. (6.5.27).

(6.5.27)\[C_{\alpha/p} = C^{*} + t_{P,N-2} \sqrt{ \frac{N-2} {\chi_{\alpha,N-2}^2} }\]

In this equation, \(C^{*}\), as defined in Eq. (6.5.28), depends on a series of calculations, outlined in Eq. (6.5.29) for \(A\), Eq. (6.5.30) for \(g\), Eq. (6.5.31) for \(h\), and Eq. (6.5.32) for \(D\).

(6.5.28)\[\begin{split}C^{*} = \begin{cases} D \, g & \text{ if } 0.5 \leq A \leq 1.5, \\ D \, h & \text{ for other values of } A \end{cases}\end{split}\]

(6.5.29)\[A=\frac{g}{h}\]

(6.5.30)\[g = \sqrt{\frac{1}{N} + \frac{(x_{min}-\bar{x})}{S_{xx}}}\]

(6.5.31)\[h = \sqrt{\frac{1}{N} + \frac{(x_{max}-\bar{x})^2}{S_{xx}}}\]

(6.5.32)\[\int_{-D}^{D} \int_{-AD}^{AD} \frac{1}{2 \pi \sqrt{ 1 - \rho^2 }} \left[ 1 + \frac{u^2 - 2 \, \rho \, u \, v + v^2 } {(N-2) (1-\rho^2)} \right] ^{\frac{-N}{2}} \, du \, dv \leq \alpha\]

The parameter \(D\) is calculated iteratively to the desired level of confidence. The parameter \(\rho\) is defined in Eq. (6.5.33), using the unweighted calculations referenced above.

(6.5.33)\[\rho = \frac{1}{g \, h} \left[ \frac{1}{N} + \frac{(x_{min}-\bar{x})(x_{max}-\bar{x})}{S_{xx}} \right]\]

6.5.3.4. `parametric` – Historical Parametric Method

The historical parametric method is suitable for data which has no statistically significant trend that is appropriately or conservatively represented by a normal distribution. This method uses the lower tolerance limit (LTL) approach to develop a bias and bias uncertainty. The bias and bias uncertainty are calculated by Eq. (6.5.34) and Eq. (6.5.35), respectively.

(6.5.34)\[\beta = \bar{k}_{norm}-1\]

(6.5.35)\[\sigma_{\beta} = KS_p\]

In Eq. (6.5.34), \(\bar{k}_{norm}\) is the either the unweighted or uncertainty-weighted average of \(k_{norm}\) in a dataset calculated by Eq. (6.5.10) or Eq. (6.5.11), respectively. The parameters required to calculate the bias uncertainty per Eq. (6.5.35) are the single-sided lower tolerance factor, \(K\), and the square root of the pooled variance, \(S_p\).

The square root of the pooled variance calculation is detailed in Eq. (6.5.36), and it comprises the variance associated with the statistical scatter of datapoints about the mean, \(\sigma^2_{\bar{k}_{norm}}\), and the average total variance associated with the individual \(k_{norm}\) values, \(\bar{\sigma}^2\).

(6.5.36)\[S_p = \sqrt{\sigma^2_{\bar{k}_{norm}} + \bar{\sigma}^2}\]

The equations for the variance associated with scatter about the mean are provided for the unweighted method in Eq. (6.5.37) and weighted method in Eq. (6.5.38).

(6.5.37)\[\sigma^2_{\bar{k}_{norm}} = \frac{\sum_{i=1}^{N}(k_{norm_i} - \bar{k}_{norm})^2}{N - 1}\]

(6.5.38)\[\sigma^2_{\bar{k}_{norm}} = \frac{ \frac{1}{N-1} \sum_{i=1}^{N} \frac{1}{\sigma_i^2} (k_{norm_i} - \bar{k}_{norm})^2} {\frac{1}{N} \sum_{i=1}^{N} \frac{1}{\sigma_i^2}}\]

The “within variance” or “average total variance,” which will be referred to as the average total variance hereinafter, described by Eq. (6.5.19) for the unweighted method and by Eq. (6.5.20) for the weighted method, combines the uncertainties associated with the input dataset.

The appropriate implementation of the single-sided lower tolerance factor, \(K\), using Eq. (6.5.39), relies on the assumption of a normal distribution being applicable or conservative.

(6.5.39)\[K = \frac{t_{\gamma,N-1,\delta}}{\sqrt{N}}\]

In this equation, \(t_{\gamma,N-1,\delta}\) is the inverse of the cumulative noncentral t-distribution corresponding to the desired confidence level, \(\gamma\), with a non-centrality parameter, \(\delta\), calculated per Eq. (6.5.40), using \(z_p\), the z-score corresponding to the desired proportion of the population, \(p\).

(6.5.40)\[\delta = z_p \sqrt{N}\]

6.5.3.5. `nonparametric` – Historical Non-Parametric Method

The historical non-parametric method is suitable for calculating bias and bias uncertainty without relying on statistical methods applicable only for normal distributions; no assumptions are made regarding the shape of the distribution for the dataset.

For this method, the bias is calculated in the same manner as that of the parametric method, per Eq. (6.5.34), with \(\bar{k}_{norm}\) calculated using either Eq. (6.5.10) for the unweighted method or Eq. (6.5.11) for the weighted method.

The bias uncertainty is calculated per Eq. (6.5.41) below. This method requires identifying \(k_{norm_r}\) and \(\sigma_{k_{norm_r}}\) corresponding to the lower rank value for the desired confidence level as well as calculating the applicable nonparametric margin, \(NPM\).

(6.5.41)\[\begin{split}\sigma_{\beta} = \begin{cases} \bar{k}_{norm} - k_{norm_r} + \sigma_{k_{norm_r}} + NPM & \text{if } \bar{k}_{norm} \leq 1, \\ 1 - k_{norm_r} + \sigma_{k_{norm_r}} + NPM & \text{if } \bar{k}_{norm} > 1, \\ \end{cases}\end{split}\]

The \(NPM\) is applied (\(NPM > 0\))for cases with less than the required number of points to justify a lower rank of one, determined by the degree of confidence calculated by Eq. (6.5.42).

(6.5.42)\[C = 1 - \sum_{j=0}^{m-1} \frac{N!}{j!(N-j)!}(1-q)^j q^{N-j}\]

In this equation, \(q\) is the desired proportion of the population (typically 0.95), \(N\) is the number of points in the dataset, and \(m\) is the rank of the data point under consideration, where the lowest \(k_{norm_i}\) value corresponds to a rank of 1 (\(m=1\)), the second-lowest corresponding to a rank of 2 (\(m=2\)), and the highest value corresponds to rank \(N\) (\(m=N\)).

If the sample size is insufficient for the desired confidence level at the lowest rank (\(m=1\))—for example, having less than 59 points for a desired confidence level of 95% using the lowest rank—then a nonparametric margin would be necessary to justify the lack of statistical confidence. Historically, \(NPM\) values based on the calculated degree of confidence for 95% of the population are as follows:

Table 6.5.2 NPM required given calculated confidence
Degree of Confidence for 95% of the population	NPM
\(>\) 90%	0.00
\(>\) 80%	0.01
\(>\) 70%	0.02
\(>\) 60%	0.03
\(>\) 50%	0.04
\(>\) 40%	0.05
\(\leq\) 40%	Need additional data

6.5.4. Available Tests

The sections below briefly describe the available tests. Two tests for normality are included: chisq, the \(\chi^2\) normality test, and anderson_darling, the Anderson–Darling normality test using the modified test statistic, a function of the sample size. One trend significance test, t-test, is available as well.

The t-test for trend significance is the recommended first step for evaluating a data set, as the existence of a significant linear trend necessitates the use of trending methods for calculating bias, bias uncertainty, and the corresponding USL. For data with no significant trend, the normality tests can be used to assess which nontrending method is suitable. For normal distributions, this is the parametric method; otherwise, the nonparametric method is suitable.

6.5.4.1. `t-test` – Trend Significance t-test

Performs the t-test for trend significance for datasets with at least 5 points using either uncertainty-weighted or unweighted methods to determine whether a significant trend exists.

The method calculates \(t_{fit}\) per Eq. (6.5.43) using parameters calculated with either the weighted or unweighted method outlined in CR6698 – Single-Sided Lower Tolerance Band.

(6.5.43)\[t_{fit} = \frac{|\beta_1|}{ \frac{ \sigma_{fit} }{ \sqrt{S_{xx}} } }\]

The calculated test statistic is compared to the \(t\)-statistic, \(t_{n-2,\frac{\alpha}{2}}\), with \(n-2\) degrees of freedom and a confidence level of \(1-\alpha\).

If \(t_{fit} > t_{n-2,\frac{\alpha}{2}}\), then the null hypothesis is rejected, and it is concluded that \(\beta_1 \neq 0\). If \(t_{fit} \leq t_{n-2,\frac{\alpha}{2}}\), the null hypothesis is not rejected, and it is concluded that \(\beta_1 = 0\).

6.5.4.2. `chisq` – \(\chi^2\) Normality Test

Performs the \(\chi^2\) goodness-of-fit test for normality per the description in “Testing for Normality” by Thode [VADER-HCT02] for datasets with at least 5 points. The test returns the \(\chi^2\) test statistic for the number of bins defined by the num_bins, which is compared to the reference \(\chi^2\) statistic corresponding to the input confidence in confidence. Setting num_bins appropriately directly influences the accuracy of the test, so the user is advised to research the test further to better understand the implications and conditions for a poorly defined number of bins.

6.5.4.3. `anderson_darling` – Anderson–Darling Normality Test

Performs the Anderson–Darling goodness-of-fit test; this calculates the test statistics \(A^{2}\) and \(A^{*2}\), which are used to generate the corresponding p-value. The set confidence level is then assessed with respect to the calculated p-value to either accept or reject the null hypothesis.

The implementation of this method uses the modified statistic described in “Goodness-of-Fit Techniques” by D’Agostino and Stephens [VADER-DAgostinoS86], where the test statistic \(A^{2}\) is modified from Eq. (6.5.44) as a function of sample size (\(n\)) per Eq. (6.5.47).

(6.5.44)\[A^{2} = -n - \frac{1}{n} \sum^{n}_{i=1} \left[ (2i -1) * \log(Z_{(i)}) + (2n+1-2i) * \log(1-Z_{(i)}) \right]\]

In Eq. (6.5.44), the \(log\) functions are base \(e\), and the values \(Z_{(i)}\) are the corresponding cumulative probability for each datapoint \(x_i\) arranged in ascending order, calculated using the sample mean in Eq. (6.5.45) and sample uncertainty in Eq. (6.5.46).

(6.5.45)\[\bar{x} = \frac{\sum_{i=1}^{n}}{n}\]

(6.5.46)\[\sigma = \sqrt{\frac{\sum_{i=1}^{n}(x_i -\bar{x})^2}{n-1}}\]

(6.5.47)\[A^{*2} = A^{2} (1.0 + \frac{0.75}{n} + \frac{2.25}{n^{2}})\]

6.5.5. Available Parameters

Table 6.5.3 shows all of the available parameters, their default value, and the methods and tests to which they apply. Any parameter can be put into the global parameters block. Only parameters that apply to a specific method or test can be put into that method or tests parameters block. Following the table is a more detailed description of each parameter.

Table 6.5.3 Available Parameters for VADER
Keyword	Default Value	Available In
tsunami	No	`USL1` `USL2`
extrapolate	Yes	`USL1`
fit_confidence	0.95	`USL1` `USL2`
admin_margin	0.05	`USL1` `USL2` `CR6698` `nonparametric` `parametric`
xrange	All X Values	`USL1` `USL2` `CR6698` `nonparametric` `parametric`
proportion_confidence	0.95	`USL2`
proportion	0.95	`USL2`
f_p	0.95	`CR6698`
normal_p	0.95	`CR6698`
chi_p	0.95	`CR6698`
weighted	No	`CR6698` `nonparametric` `parametric` `t-test`
confidence	0.95	`nonparametric` `parametric` `chisq` `t-test` `anderson_darling`
proportion_of_pop	0.95	`nonparametric` `parametric`
use_npm	Yes	`nonparametric`
num_bins	5	`chisq`

tsunami: This enables a TSUNAMI-mode calculation for VADER, requiring that all X values be between -1.0 and 1.0 for the method calculation. This also minimally tailors the output printing for the given constraints on the data.
extrapolate: This allows extrapolation past the data values entered by the user. This is generally only applicable when the user has input values in the trend_values block.
fit_confidence: Confidence in the statistical result. Sometimes referred to as \(1 - \gamma\). In USL1 this is used to calculated the t-statistic quantile.
admin_margin: Administrative margin subtracted from the result. This can be used to add an extra layer of confidence that the value reported is below criticality.
xrange: Allows the user to tailor X values for each method or test or change the values for all tests without having to change the data block. Data is entered as an array of two values. By default all of the X values are used.
proportion_confidence: In USL2 this controls the confidence on the proportion that is in the confidence band.
proportion: In USL2 this controls the proportion of the data points that will be inside the confidence band.
f_p: The percentile used for the F-distribution in the single-sided lower tolerance band method.
normal_p: The percentile used for the normal distribution in the single-sided lower tolerance band method.
chi_p: The percentile used for the \(\chi^2\) distribution in the single-sided lower tolerance band method.
weighted: Whether to use the uncertainty-weighted calculations.
confidence: Statistical confidence that the proportion calculated to be above the cutoff is correct.
proportion_of_pop: Proportion of the population that will be above the cutoff value in the non-parametric method.
use_npm: Use (add) non-parametric margin to the non-parametric method.
num_bins: Number of bins to use for the calculation of the \(\chi^2\) test statistic.

6.5.6. Example Trending Bias Assessment

This section illustrates the bias assessment workflow using VADER. Consider the following VADER input for the U-233 critical experiments and their corresponding energy of the average lethargy of neutrons causing fission (EALF) data sourced from Example 1 in the Appendix of NUREG/CR-7311 [VADER-CMM+25]. Note that in this example, the EALF is not expected to be linear, but the natural logarithm of the EALF is expected to be linear. Thus, ln(EALF) is used for the trending parameter X, which corresponds to the U-233 C/E Y values. It is assumed in this input that a trending analysis will be applicable for this data set, but that assumption should be confirmed by evaluating trend significance. In this example, two trending methods are used: the CR6698 method and the USL1 method. One test is used: the t-test for trend significance. Three ln(EALF) trend values of interest are also specified: -2.996, 0.0, and 3.912, which correspond to EALF values of 0.01, 1.0, and 50.0 eV. Default parameters are used for each method and test.

Example 6.5.7 VADER Trending Data Example Input

=vader
 title="U-233 Critical Experiments"

 data = [
 ' Example 1 Data - U-233 Critical Experiments
 '   ln(EALF)     C/E   uncertainty of C/E
 0.43178241643 1.00082 0.00250
 -0.24590053844 1.00227 0.00241
 -0.76787072676 1.00058 0.00250
 1.92570744174 0.98522 0.00818
 2.08069076108 0.98053 0.00834
 2.15408508468 0.98126 0.00648
 1.31372366829 0.99217 0.00605
 2.22137503757 0.98462 0.00807
 1.45628673294 0.98553 0.00601
 2.26695791535 0.98174 0.00579
 1.51512723296 0.98015 0.00549
 1.99470031322 0.97978 0.00666
 2.31253542385 0.97862 0.00519
 2.04898233420 0.98018 0.00559
 1.49514876603 0.98123 0.00893
 1.62727783056 0.98207 0.00697
 1.69927861643 0.98000 0.00735
 0.93609335917 0.98848 0.00544
 1.76644166124 0.97843 0.00558
 1.80828877118 0.97523 0.00810
 1.10194007876 0.97965 0.00549
 1.85002837735 0.97295 0.00487
 1.87333945622 0.97805 0.00479
 1.55180879960 0.99000 0.00465
 0.68813463874 0.99236 0.00804
 0.82417544297 0.98533 0.00798
 0.87546873735 0.98907 0.00643
 0.94000725849 0.98357 0.00600
 0.97832612279 0.97727 0.00958
 0.99325177301 0.99085 0.00704
 1.03318448335 0.97571 0.00517
 0.73236789371 0.99382 0.00457
 0.83290912294 0.98998 0.00515
 0.59332684528 0.97444 0.00273
 0.37843643572 0.97763 0.00518
 0.26236426447 0.98645 0.00671
 0.31481073984 0.98620 0.00543
 0.35065687161 0.97657 0.00967
 0.38526240079 0.97311 0.00652
 0.40546510811 0.96863 0.00484
 -3.23398946268 1.00141 0.00311
 -3.21887582487 1.00067 0.00330
 -3.20398721237 1.00012 0.00330
 -3.18931702263 1.00103 0.00331
 -3.17485893945 1.00025 0.00330
 -1.74869997977 1.00187 0.00868
 -2.00991547903 0.98990 0.00858
 -2.25379492882 1.00592 0.00872
 -2.46510402249 1.00252 0.00869
 -2.60233974668 1.00738 0.00873
 -2.71810053696 0.99400 0.00861
 -2.77258872224 0.98375 0.00853
 -2.85597033118 0.99783 0.00865
 -2.95843648881 0.98647 0.00855
 -2.99373427089 0.99937 0.00866
 -3.06401111431 1.00803 0.00874
 -1.29828348380 0.98872 0.00857
 -0.71743987313 0.98852 0.00857
 -1.97328134585 0.99754 0.00864
 -2.34758245892 1.00412 0.00870
 -2.91139112512 1.00640 0.00872
 -1.15518264016 1.00214 0.00873
 -1.04982212450 1.01754 0.01541
 -1.09064411902 0.99854 0.00869
 -0.23825718912 1.00206 0.01259
 0.05826890812 1.00933 0.01233
 -2.04022082853 1.02084 0.00888
 -2.48051630149 1.01405 0.00882
 -2.67220054821 1.01003 0.00878
 -2.78062089394 1.00984 0.00878
 -3.07477548089 1.00751 0.00877
 -1.76026080217 0.99887 0.00876
 -2.01740615076 1.00241 0.00859
 -1.29828348380 0.99177 0.00879
 -0.71539278951 0.98154 0.00869
 -0.95191790952 0.98867 0.00887
 -0.72154665508 1.00145 0.01049
 -0.95972028980 0.99817 0.01036
 -1.98050159382 1.00466 0.01023
 -2.78223509929 1.00160 0.00401
 -2.90955457731 1.00456 0.00492
 -3.29144651770 1.00089 0.00290
 -3.27280416689 0.99933 0.00441
 -3.28341434601 1.00118 0.00401
 -3.29413830937 1.00163 0.00381
 -3.30497852392 0.99981 0.00380
 0.22314355131 0.99022 0.00505
 -1.75446368448 1.00010 0.00280
 -1.79576749063 1.00004 0.00250
 -1.91054300522 1.00998 0.00233
 -2.22562405186 1.00247 0.00151
 -2.39250980052 1.00412 0.00713
 -2.51825662947 1.00523 0.00111
 -2.91323105204 1.00126 0.00381
 -2.91692109313 0.99854 0.00479
 -1.85150947363 1.00548 0.00735
 -1.85150947363 1.00557 0.00705
 -1.84516024596 1.00562 0.00695
 -1.83885107676 1.00595 0.00735
 -1.83258146375 1.00653 0.00675
 -1.89047544217 1.00647 0.00504
 -1.88387475814 1.00629 0.00544
 -1.88387475814 1.00681 0.00504
 -1.87731735759 1.00707 0.00454
 -1.87731735759 1.00769 0.00464
 -1.89711998489 1.00529 0.00543
 -1.88387475814 1.00621 0.00504
 -1.89047544217 1.00372 0.00623
 -1.87731735759 1.00642 0.00514
 -2.25379492882 1.02100 0.00787
 -2.42136862898 0.99338 0.00686
 -2.44992568089 0.99596 0.00518
 -2.51949809456 1.00020 0.00200
 -2.51701670371 0.99634 0.00887
 -2.76303450943 0.99820 0.00559
 -2.86998106825 1.00238 0.00341
 0.09531017980 0.98999 0.00743
 0.21511137962 0.98558 0.00690
 -0.32711614170 0.98933 0.00406
 -0.22941316433 0.98653 0.00691
 0.12221763272 0.98994 0.00505
 -0.36672527979 0.99315 0.00745
 -0.26526847761 0.99395 0.00686
 -0.21443161071 0.99177 0.00684
 -0.76356964486 0.99644 0.00359
 -0.16605458433 0.98968 0.00594
 -0.14156356432 0.98839 0.00425
 -0.68319684971 0.99653 0.00289
 -0.10758521068 0.97430 0.00546
 -0.09211528891 0.97484 0.00507
 -1.24132859087 0.99483 0.00786
 -1.15518264016 0.99794 0.00699
 -1.10866262452 0.99612 0.00618
 -1.07002483182 0.99407 0.00547
 -1.04696905552 0.99084 0.00505
 -1.49165487678 0.99585 0.00229
 -2.06356819252 0.99396 0.00656
 -2.03255795578 0.99871 0.00629
 -2.01740615076 0.99682 0.00578
 -2.00248050054 0.99532 0.00508
 -1.98777435315 0.99465 0.00478
 -1.98050159382 0.99381 0.00547
 -1.23443201181 1.00439 0.00372
 -1.23100147671 1.00577 0.00443
 -1.23100147671 1.00458 0.00362
 -1.23443201181 1.00600 0.00353
 -1.23443201181 0.99620 0.00339
 -1.23443201181 0.99530 0.00338
 -1.23443201181 0.99461 0.00278
 -1.23443201181 0.99569 0.00269
 -1.24479479885 1.00389 0.00301
 -1.24479479885 1.00456 0.00412
 -1.24479479885 1.00485 0.00473
 -1.93102153656 1.00505 0.00362
 -1.93102153656 1.00482 0.00262
 -1.93102153656 1.00582 0.00272
 -1.92414865727 1.00965 0.00313
 -1.91732269220 0.99429 0.00279
 -1.92414865727 0.99588 0.00359
 -1.92414865727 0.99486 0.00348
 -1.93794197941 1.00902 0.00283
 -1.93794197941 1.00882 0.00343
 -1.93794197941 1.00895 0.00313
 -1.93794197941 1.00839 0.00242
 -2.50592601701 1.00106 0.00401
 -2.50592601701 1.00679 0.00343
 -2.50715225874 1.00384 0.00372
 -2.50838000597 0.99960 0.00371
 -2.50838000597 1.00033 0.00311
 -2.50838000597 0.99923 0.00320
 -2.85771097566 1.01071 0.00344
 -2.85771097566 1.01307 0.00325
 -2.85771097566 1.01283 0.00396
 -2.16282315062 1.00454 0.00322
 -2.18925640769 1.00051 0.00250
 -2.21640739675 1.00519 0.00352
 -2.48650793115 1.00581 0.00403
 -2.50715225874 1.00240 0.00291
 -2.91139112512 1.00021 0.00290
 -2.89317568523 0.99990 0.00370
 ]

 trend_values=[ -2.996 0.0 3.912 ]

 parameters{extrapolate=no}

 methods{
 CR6698{}
 USL1{}
 }

 tests{
 t-test{}
 }

 end

Before using any results of a method, confirm that the method is suitable by investigating the results of the applicable test. In this case, the t-test evaluates the null hypothesis that no trend exists, and the output for this example states “Null Hypothesis Rejected,” meaning there is a significant trend in the data. Based on this result, it is appropriate to proceed to the trending methods’ output. The following output should be seen for the t-test.

From this output, it can be confirmed that the null hypothesis was rejected, meaning there is a statistically significant trend based on this t-test. Because there is a statistically significant trend, it is likely appropriate to proceed with the use of trending methods for the USL calculation.

The first method output shown is for the CR6698 single-sided lower tolerance band method. Information on the dataset and parameters is printed at the start of the output. Notably, a default administrative margin of 0.05 is applied across all methods and is incorporated in the USL calculation. For this example, extrapolation is disabled.

The “Method Parameters” section should be used to verify expectations for the data set and parameters before proceeding to the results. In the “Method Results” section, the output data includes the pooled standard deviation (\(S_p\)), the sum of squared deviations from the mean for X values (\(S_{xx}\)), the linear fit intercept (\(\beta_0\)), the linear fit slope (\(\beta_1\)), and the fit trend value, where \(k=1\). The “Trend Values” output section will produce the USL as a function of trend values spanning the data X values, and the values specified explicitly in trend_values are included in the output table as well. For each trend value, the following output values are printed: the trended k-eff \(k(x)\) based on the linear regression; the adjusted \(k(x)\), which caps at 1.0; the method-specific bias uncertainty of the trend \(\sigma(x)\); administrative margin; and USL.

Note that extrapolation has not been enabled globally; here, the extrapolate parameter applies only for the USL1 method. Therefore, it is expected to see extrapolated trend values in the output of the CR6698 method but not in the USL1 method. In the USL1 method, any trend value outside of the range of the X values in the data set will return “N/A.” The output of the USL-1 method is included below.

Example 6.5.10 VADER Tending Example USL-1 Method Output

 Method (2 of 2): USL1
 =====================
 Trend Name: USL Method 1 (Confidence Band with Administrative Margin)

 Number of data points:                            180
 Data Set Minimum X value:                         -3.305
 Data Set Maximum X value:                         2.3125
 Data Set unweighted X average:                    -1.1994
 Data Set unweighted Y average:                    0.99661

 -----------------
 Method Parameters
 -----------------
 TSUNAMI-Mode:                                     No
 Extrapolation On:                                 No
 Confidence [%]:                                   0.95
 Administrative Margin:                            0.05
 Minimum X Value:                                  -3.30498(DATA_MIN)
 Maximum X Value:                                  2.31254(DATA_MAX)

 --------------
 Method Results
 --------------
 Pooled std. dev.:                                 0.00945737
 Within variance:                                  3.86665e-05
 Variance of fit:                                  5.07754e-05
 W:                                                0.015905
 S_xx:                                             425.815
 Fit Coeff. 0 (Intercept):                         0.99064
 Fit Coeff. 1 (Slope)::                            -0.00497895

 ------------
 Trend Values
 ------------
 Equation:
            0.924735 + (-0.004979)*X (X >  -1.879985)
            0.934095                (X <= -1.879985)

     Trend Value         k(x)    Adj. k(x) Bias Unc.(x)  Adm. Margin          USL
        -3.30498      1.00709      1.00000      0.01590      0.05000      0.93410
        -2.99600      1.00556      1.00000      0.01590      0.05000      0.93410
        -2.50248      1.00310      1.00000      0.01590      0.05000      0.93410
        -1.87998      1.00000      1.00000      0.01590      0.05000      0.93410
        -1.69997      0.99910      0.99910      0.01590      0.05000      0.93320
        -0.89747      0.99511      0.99511      0.01590      0.05000      0.92920
        -0.09497      0.99111      0.99111      0.01590      0.05000      0.92521
         0.00000      0.99064      0.99064      0.01590      0.05000      0.92473
         0.70753      0.98712      0.98712      0.01590      0.05000      0.92121
         1.51003      0.98312      0.98312      0.01590      0.05000      0.91722
         2.31254      0.97913      0.97913      0.01590      0.05000      0.91322
         3.91200          N/A          N/A          N/A          N/A          N/A

 NOTE: N/A is printed in the table if extrapolation is off and a value outside of the range was desired.

The “Method Parameters” output section confirms that the extrapolation parameter is both part of the method and is disabled. This is reflected in the “Trend Values” output section, which prints “N/A” for trend values at 50 eV (ln(EALF) = 3.912), which is a value outside of the X value range, as indicated by the minimum and maximum X values printed near the start of the method output. In the USL-1 method, the bias uncertainty as a function of x is constant across the values in-range. This is a conservatism in the USL-1 methodology that uses the maximum of \(\sigma_{\beta}(x)\) evaluated over the X range. When extrapolation is enabled, the trended value for bias uncertainty is used outside of the X range. This would correspond to a bias uncertainty of 0.01615 at the trend value corresponding to 50 eV.

References

[VADER-CMM+25] (1,2,3,4)

J. B. Clarity, W. J. Marshall, D. E. Mueller, S. S. Powers, B. T. Rearden, S. M. Bowman, and A. Barto. Determination of bias and bias uncertainty for criticality safety computational methods. Technical Report NUREG/CR-7311, Oak Ridge National Laboratory, Oak Ridge, TN (USA), 2025.

[VADER-DAgostinoS86]

Ralph B. D'Agostino and Michael A. Stephens. Goodness-of-Fit Techniques. Marcel Dekker, Inc., 1986. ISBN 9780367580346. doi:https://doi.org/10.1201/9780203753064.

[VADER-DRWT01] (1,2)

J. C. Dean and Jr. R. W. Tayloe. Guide for validation of nuclear criticality safety calculational methodology. Technical Report NUREG/CR-6698, Prepared for the US Nuclear Regulatory Commission by Science Applications International Corporation, Oak Ridge, TN (USA), 1 2001.

[VADER-HCT02]

Jr. Henry C. Thode. Testing for Normality. Marcel Dekker, Inc., 2002. ISBN 9780429213250. doi:https://doi.org/10.1201/9780203910894.

6.5. VADER: Trending Analysis for Code/Data Validation

6.5.1. Input

6.5.1.1. Legacy Input

6.5.2. Detailed Input

6.5.2.1. data

6.5.2.2. title

6.5.2.3. parameters

6.5.2.4. trend_values

6.5.2.5. methods

6.5.2.6. tests

6.5.3. Available Methods

6.5.3.1. CR6698 – Single-Sided Lower Tolerance Band

6.5.3.2. USL1 – Confidence Band with Administrative Margin

6.5.3.3. USL2 – Single-Sided Uniform Width Closed Interval Approach

6.5.3.4. parametric – Historical Parametric Method

6.5.3.5. nonparametric – Historical Non-Parametric Method

6.5.4. Available Tests

6.5.4.1. t-test – Trend Significance t-test

6.5.4.2. chisq – \(\chi^2\) Normality Test

6.5.4.3. anderson_darling – Anderson–Darling Normality Test

6.5.5. Available Parameters

6.5.6. Example Trending Bias Assessment

6.5.2.1. `data`

6.5.2.2. `title`

6.5.2.3. `parameters`

6.5.2.4. `trend_values`

6.5.2.5. `methods`

6.5.2.6. `tests`

6.5.3.1. `CR6698` – Single-Sided Lower Tolerance Band

6.5.3.2. `USL1` – Confidence Band with Administrative Margin

6.5.3.3. `USL2` – Single-Sided Uniform Width Closed Interval Approach

6.5.3.4. `parametric` – Historical Parametric Method

6.5.3.5. `nonparametric` – Historical Non-Parametric Method

6.5.4.1. `t-test` – Trend Significance t-test

6.5.4.2. `chisq` – \(\chi^2\) Normality Test

6.5.4.3. `anderson_darling` – Anderson–Darling Normality Test