6.5. VADER: Trending Analysis for Code/Data Validation

S. Hart, J. Clarity

VADER (Validation Analysis Data Evaluation Resource) is a new module in SCALE 6.3 derived from the legacy USLSTATS program. VADER is a tool that allows the determination of bias and bias uncertainty for criticality safety computational methods. The older USLSTATS program, written in Java, existed outside of SCALE and provided tools to calculate only the confidence band with administrative margin (sometimes called USL-1) and the single-sided uniform width closed interval (USL-2). For normality testing it offered only a crude chi-squared normality test that had no user-configurable options and presented a simple pass/no-pass functionality.

VADER has been completely written in C++ using modern coding practices and standards. It has been moved to be part of the SCALE code package and is run through the normal SCALE executables. To support the VADER methods, large updates were made to the statistical capabilities of SCALE. Interfaces to known statistical distributions were added and many of the statistical calculations required for VADER were made available to all SCALE sequences. In addition to being able to process legacy USLSTATS inputs, VADER adds support for the single-sided tolerance band method (from NUREG/CR-6698) as well as two non-trending methods: the “historical” nonparametric and parametric methods, also as described in NUREG/CR-6698. VADER also includes several new tests for normality in addition to adding user control options on the original chi-squared normality test. A new t-test for the statistical significance of the trend is included.

VADER has also been integrated into the Fulcrum GUI used by SCALE. This allows VADER to benefit from the auto-complete, templating, and input validation features offered by Fulcrum. In addition, Fulcrum can plot the results given by VADER, and the plot files have been expanded to include the new methods available in VADER.

6.5.1. Theory

The methods and theory for VADER are largely taken from NUREG/CR-6698. That document contains equations and descriptions of the methods used in VADER. It also includes working examples, some of which data was used to verify the results and create example problems for VADER.

6.5.2. Input

The VADER input is based on the SON input syntax common in other modernized SCALE codes such as ORIGEN and ORIGAMI. The SON input format consists of key/value pairs where the value can be either a single item, an array of items, or other SON blocks.

The VADER input generally just has one level of depth. There are no SON blocks inside of SON blocks. All input is mostly arrays at the top level of the tree. A quick look at all the top level items is given in Table 6.5.1. Each top level input will then be described in detail in the next section.

Table 6.5.1 Summary of top level options for VADER

Keyword	Required	Description
data	Yes	Data entered in triplets in the form X, Y, std. dev.
title	No	Title to give the case.
parameters	No	Global parameters that apply to each test or trend.
trend_values	No	User desired trend values to run the tests on.
methods	Yes	Methods that the user wishes to apply to the data.
tests	No	Statistical checks that the user wishes to apply to the data.

6.5.2.1. Legacy Input

For backwards compatibility VADER supports reading the original USLSTATS format. This format is restricted to USL methods USL1 and USL2. Results should match those obtained using the Java-based USLSTATS program with some exceptions detailed below.

• The extrapolation parameter has been updated to be more rigorous. Only USL1 can do extrapolation and, if turned off, values outside of the range will be labeled in the table as N/A. USL2 will always give N/A for values outside of the range.

• The USL1 method one will always use \(W\) inside of the range and then the larger of \(W\) or \(w(x)\) outside of the range (if extrapolating). This differs from legacy USLSTATS that used w(x) across the entire range if extrapolating.

6.5.3. Detailed Input

6.5.3.1. data

The data input array is a flat array consisting of data triplets. There is no punctuation between the triplets. The data is entered in the order \(X_1 \, Y_1 \, \sigma_1 \; X_2 \, Y_2 \, \sigma_2 \; ... \; X_N \, Y_N \, \sigma_N\).

Where

X	The trended value, usually something like enrichment or moderator density.
Y	The y value, usually something like C/E or eigenvalue.
\(\sigma\)	The standard deviation of Y.

VADER will emit an error if the number of entries in this array is not divisible by three.

Example 6.5.1 Snippet of data input

data=[2.4 0.9976 0.00122 2.6 0.9978 0.00022
      ...
      1.9 1.0011 0.00213 1.99 1.013 0.00012]

6.5.3.2. title

The title entry allows the user to specify a title for the case. The title is cosmetic and doesn’t affect any of the results. The title is echoed to the output in several places and is included on some of the graphs generated by VADER.

Example 6.5.2 title

title="My Very First VADER Case"

6.5.3.3. parameters

The parameters section can exist outside of any test or method. If a parameter is listed in this global section, then it will apply to any method or test that has that parameter available instead of the default value. Values in this global parameters section can be overridden in the specific method or test block for which the user wishes to override the global parameter. For a list of all parameters, their default values, and which tests and methods they are applied to, see Sect. 6.5.6.

Example 6.5.3 parameters array

parameters {extrapolate=yes admin_margin=0.03}

6.5.3.4. trend_values

This array allows the user to insert values at which they wish the trend Y (typically the upper subcritical limit) values to be calculated. Extrapolation outside of the data range is only available if the method supports it and extrapolation is enabled.

Example 6.5.4 trend_values array

trend_values=[-0.1 0.2 2.4 10.0]

6.5.3.5. methods

This is where the user specifies the methods to enable and, optionally, gives method-specific parameters. Each method is followed by an array of the parameters the user wishes to set. Any omitted parameters are initialized with the default value (see Table 6.5.2). If the user wishes to enable a method with all default values, then an empty array can be given.

Example 6.5.5 methods block with some default methods and some specific parameters

methods {
  USL1 {} % uses default parameters
  CR6698 {weighted=yes admin_margin=0.03 chi_p=0.97}
  parametric {confidence=0.9}
}

6.5.3.6. tests

This block is similar to the methods block, but it allows the user to specify which tests they wish to enable. These tests are run on the data and generally either pass or fail and give a few statistical values as their result. Most of the tests implemented for VADER for now are normality tests.

Example 6.5.6 tests block

tests {
  chisq {} % default parameters
  t-test {confidence=0.98}
}

6.5.4. Available Methods

The sections below indicate the input name to be used to activate the available method. Details on the methods can currently be found in NUREG/CR-6698.

6.5.4.1. USL1 - Confidence Band with Administrative Margin

6.5.4.2. USL2 - Single-Sided Uniform Width Closed Interval Approach

6.5.4.3. CR6698 - Single-Sided Lower Tolerance Band

6.5.4.4. nonparametric - Historical Non-Parametric Method

6.5.4.5. parametric - Historical Parametric Method

6.5.5. Available Tests

The sections below indicate the input name to be used to activate the available test. Details on the tests can currently be found in NUREG/CR-6698.

6.5.5.1. chisq - \(\chi^2\) Normality Test

6.5.5.2. t-test - Trend Significance t-test

6.5.5.3. anderson_darling - Anderson-Darling Normality Test

6.5.6. Available Parameters

Table 6.5.2 shows all of the available parameters, their default value, and the methods and tests to which they apply. Any parameter can be put into the global parameters block. Only parameters that apply to a specific method or test can be put into that method or tests parameters block. Following the table is a more detailed description of each parameter.

Table 6.5.2 Available Parameters for VADER

Keyword	Default Value	Available In
tsunami	Off	USL1 USL2
extrapolate	On	USL1
fit_confidence	0.95	USL1 USL2
admin_margin	0.05	USL1 USL2 CR6698 nonparametric parametric
xrange	All X Values	USL1 USL2 CR6698 nonparametric parametric
proportion_confidence	0.95	USL2
proportion	0.95	USL2
f_p	0.95	CR6698
normal_p	0.95	CR6698
chi_p	0.95	CR6698
weighted	No	CR6698 nonparametric parametric
confidence	0.95	nonparametric parametric
proportion_of_pop	0.95	nonparametric parametric
use_npm	Yes	nonparametric

6.5.6.1. tsunami

This enables a TSUNAMI-mode calculation for VADER. It requires that all X values be between -1.0 and 1.0.

6.5.6.2. extrapolate

This allows extrapolation past the data values entered by the user. This is generally only applicable when the user has input values in the trend_values block.

6.5.6.3. fit_confidence

Confidence in the statistical result. Sometimes referred to as \(1 - \gamma\). In USL1 this is used to calculated the t-statistic quantile.

6.5.6.4. admin_margin

Administrative margin subtracted from the result. This can be used to add an extra layer of confidence that the value reported is below criticality.

6.5.6.5. xrange

Allows the user to tailor X values for each method or test or change the values for all tests without having to change the data block. Data is entered as an array of two values. By default all of the X values are used.

Example 6.5.7 xrange

xrange=[-0.5 2.5]

6.5.6.6. proportion_confidence

In USL2 this controls the confidence on the proportion that is in the confidence band.

6.5.6.7. proportion

In USL2 this controls the proportion of the data points that will be inside the confidence band.

6.5.6.8. f_p

The percentile used for the F-distribution in the single-sided lower tolerance band method.

6.5.6.9. normal_p

The percentile used for the normal distribution in the single-sided lower tolerance band method.

6.5.6.10. chi_p

The percentile used for the \(\chi^2\) distribution in the single-sided lower tolerance band method.

6.5.6.11. weighted

Whether to use the weighted version of the mean and standard deviation equations when calculating the statistics for this method.

6.5.6.12. confidence

Statistical confidence that the proportion calculated to be above the cutoff is correct.

6.5.6.13. proportion_of_pop

Proportion of the population that will be above the cutoff value in the non-parametric method.

6.5.6.14. use_npm

Use (add) non-parametric margin to the non-parametric method.