Validation and QC

In a regulated study, a number is not finished when one program computes it. It is finished when a second, independent program computes the same number and PROC COMPARE finds no difference. This is called double programming. In this chapter you write both programs and compare them. You see what a difference looks like, and you check a log the way a reviewer does.

15.1 The production program

The production programmer derives four ADSL variables from SDTM: the age group, the dates of first and last exposure, and the treatment duration in days. The specification for this chapter is short on purpose: “TRTEDT is the latest EX end date”. Section 15.4 shows what that short rule misses.

15_1_production.sas

/* Chapter 15: production version of four ADSL variables */
PROC SORT DATA=sdtm.ex(KEEP=usubjid exstdtc exendtc) OUT=ex; BY usubjid exstdtc; RUN;

DATA prod_trt(KEEP=usubjid trtsdt trtedt);
  SET ex;
  BY usubjid;
  RETAIN trtsdt trtedt;
  IF FIRST.usubjid THEN DO;
    trtsdt = INPUT(exstdtc, ?? E8601DA.);
    trtedt = .;
  END;
  IF NOT MISSING(exendtc) THEN trtedt = MAX(trtedt, INPUT(exendtc, ?? E8601DA.));
  IF LAST.usubjid THEN OUTPUT;
  FORMAT trtsdt trtedt DATE9.;
RUN;

DATA prod;
  MERGE sdtm.dm(IN=indm KEEP=usubjid age armcd WHERE=(armcd NE "Scrnfail")) prod_trt;
  BY usubjid;
  IF indm;
  LENGTH agegr1 $5;
  IF      MISSING(age) THEN agegr1 = "";
  ELSE IF age < 65     THEN agegr1 = "<65";
  ELSE IF age <= 80    THEN agegr1 = "65-80";
  ELSE                      agegr1 = ">80";
  IF NOT MISSING(trtsdt) AND NOT MISSING(trtedt) THEN trtdurd = trtedt - trtsdt + 1;
  DROP armcd;
RUN;

What you should see: NOTE: The data set WORK.PROD has 254 observations and 6 variables. One record for each randomised subject.

15.2 The QC program: same specification, different code

The QC programmer reads the same specification but writes the program alone. The QC program uses a different technique on purpose: PROC SQL instead of BY groups and RETAIN. When two different methods give the same answer, the answer is much more likely to be right.

15_2_qc.sas

/* Chapter 15: an independent QC version, written with PROC SQL */
PROC SQL;
  CREATE TABLE qc AS
    SELECT d.usubjid, d.age,
           CASE WHEN d.age < 65  THEN "<65"
                WHEN d.age <= 80 THEN "65-80"
                WHEN d.age > 80  THEN ">80"
                ELSE "" END                               AS agegr1 LENGTH=5,
           MIN(INPUT(e.exstdtc, E8601DA.))                AS trtsdt FORMAT=DATE9.,
           MAX(CASE WHEN NOT MISSING(e.exendtc)
                    THEN INPUT(e.exendtc, E8601DA.) ELSE . END) AS trtedt FORMAT=DATE9.,
           CASE WHEN CALCULATED trtedt NE . AND CALCULATED trtsdt NE .
                THEN CALCULATED trtedt - CALCULATED trtsdt + 1
                ELSE . END                                AS trtdurd
      FROM sdtm.dm(WHERE=(armcd NE "Scrnfail")) AS d
           LEFT JOIN sdtm.ex AS e
           ON d.usubjid = e.usubjid
      GROUP BY d.usubjid, d.age
      ORDER BY d.usubjid;
QUIT;

What you should see: NOTE: Table WORK.QC created, with 254 rows and 6 columns. and no other NOTE. Each CASE has an ELSE. Without it, SAS writes NOTE: A CASE expression has no ELSE clause., and a log check would flag the program, although the result is right.

A lesson from writing this program

The first version used INPUT(e.exendtc, E8601DA.) directly. Some EX records have no end date, so the log said NOTE: Invalid argument to function INPUT. Missing values may be generated. In a DATA step you would add ??. But PROC SQL does not accept ??: it writes ERROR 22-322: Expecting a format name. In SQL, skip the empty values with CASE WHEN NOT MISSING(...) THEN ... END.

15.3 Compare the two

15_3_compare.sas

/* Chapter 15: compare production with QC */
PROC COMPARE BASE=prod COMPARE=qc;
  ID usubjid;
RUN;
%PUT NOTE: SYSINFO after the compare = &SYSINFO.;

/* A QC program with one mistake: age 65 goes into the wrong group */
DATA qc_bad;
  SET qc;
  IF age = 65 THEN agegr1 = "<65";
RUN;

PROC COMPARE BASE=prod COMPARE=qc_bad MAXPRINT=10;
  ID usubjid;
  VAR agegr1;
RUN;
%PUT NOTE: SYSINFO after the second compare = &SYSINFO.;

/* Our production against the CDISC reference ADSL */
PROC COMPARE BASE=refadm.adsl(KEEP=usubjid agegr1 trtsdt trtedt trtdur)
             COMPARE=prod(RENAME=(trtdurd=trtdur)) MAXPRINT=10;
  ID usubjid;
  VAR agegr1 trtsdt trtedt trtdur;
RUN;

What you should see for the first compare:

Number of Observations with All Compared Variables Equal: 254.
NOTE: No unequal values were found. All values compared are exactly equal.

and in the log NOTE: SYSINFO after the compare = 0. &SYSINFO is 0 when the data sets agree, and a program can test it (9.2 lists the codes).

Now the second compare: a QC program with one mistake. It puts age 65 into the group <65.

What you should see:

USUBJID	agegr1 in production	agegr1 in QC
01-707-1206	65-80	<65
01-709-1309	65-80	<65
01-715-1107	65-80	<65
01-715-1319	65-80	<65

Number of Observations with Some Compared Variables Unequal: 4. and SYSINFO = 4096, which means “some values differ”. Four subjects are exactly 65. A boundary error hides in a few records. That is why a full compare beats a look at a frequency table.

Exercise 15-1. Change the QC program so that it computes trtdurd with the DATA step instead of SQL. Does the compare still pass?

Exercise 15-2. Add CRITERION=1E-8 to a compare of two numeric variables computed in different ways. When do you need it?

15.4 Compare with the reference ADaM

The third compare puts your production next to the reference ADSL from CDISC.

What you should see: AGEGR1 and TRTSDT agree for all 254 subjects. TRTEDT and TRTDUR differ for 6 subjects, and two of them are missing in your data:

USUBJID	TRTEDT reference	TRTEDT yours
01-704-1233	14JUL2013	04APR2013
01-705-1018	12JUL2013	.
01-705-1031	11MAY2014	18DEC2013
01-705-1303	02JUN2014	30DEC2013
01-705-1377	07MAR2014	25JAN2014
01-705-1382	13MAY2013	.

This is a real finding, and it is how QC works in practice. The production and the QC program agree, because both follow the same short specification. Double programming checks the code, not the specification. The compare with the reference finds the gap: these are the 6 subjects whose last dose record has no end date, an open dose record.

You do not decide the rule yourself. You write the finding down and ask the statistician. In the pilot, the answer is the rule of section 9.3. An open last dose record means that the subject was still dosed at the end of the study. So TRTEDT is the end of study date, RFENDTC. With that rule, 09_1_adsl.sas gives SYSINFO=0 against the reference. Chapter 16 shows how to find such a gap before QC does.

Exercise 15-3. Write a check that lists the subjects behind the 6 differences with the reference ADSL, with all their EX records.

The compare also lists attribute differences: the reference has labels, your data sets do not. In a real ADaM, labels come from the specification and must match too.

15.5 Check the log like a reviewer

A program can give the right numbers and still have a log a reviewer rejects. This macro reads a log file and counts the messages that matter:

15_4_logcheck.sas

/* Chapter 15: a log check, the same idea as the Suspicious Log */
%MACRO logcheck(file=);
  DATA _lc;
    INFILE "&file." TRUNCOVER;
    INPUT line $CHAR256.;
    LENGTH kind $45;
    /* the message words are built from two parts, so this program's own source never matches */
    IF      line =: CATS("ERR", "OR")                              THEN kind = "ERROR";
    ELSE IF line =: CATS("WAR", "NING")                            THEN kind = "WARNING";
    ELSE IF INDEX(line, "is uninitialized")                        THEN kind = "NOTE: uninitialized";
    ELSE IF INDEX(line, "Missing values were generated")           THEN kind = "NOTE: missing values generated";
    ELSE IF INDEX(line, "repeats of BY values")                    THEN kind = "NOTE: repeats of BY values";
    ELSE IF INDEX(line, "remerging summary statistics")            THEN kind = "NOTE: remerging";
    ELSE IF INDEX(line, "values have been converted")              THEN kind = "NOTE: type conversion";
    ELSE IF INDEX(line, "Invalid data")                            THEN kind = "NOTE: invalid data";
    ELSE IF INDEX(line, "Division by zero")                        THEN kind = "NOTE: division by zero";
    IF NOT MISSING(kind);
  RUN;
  TITLE "Log check of %SCAN(&file., -1, /)";
  PROC FREQ DATA=_lc;
    TABLES kind / NOCUM NOPERCENT;
  RUN;
  TITLE;
%MEND logcheck;

/* Write the log of a step with problems to a file, then check that file */
FILENAME demolog "%SYSFUNC(PATHNAME(work))/demo.log";
PROC PRINTTO LOG=demolog NEW; RUN;

DATA demo;
  SET sdtm.dm(OBS=3);
  siteid_num = siteid + 0;
  bmi        = weight / height**2;
RUN;

PROC PRINTTO LOG=LOG; RUN;

%logcheck(file=%SYSFUNC(PATHNAME(work))/demo.log)

What you should see:

kind	Frequency
NOTE: missing values generated	1
NOTE: type conversion	1
NOTE: uninitialized	2

The macro builds the words ERROR and WARNING from two parts with CATS("ERR", "OR"). If it wrote them in one piece, the program’s own source code in the log would count as a problem.

See it in DataStep Studio

The Suspicious Log tab does this check live, after every F4. It lists every ERROR and WARNING and the NOTEs above, each with a link to the program line that caused it. Use the macro for batch runs and for the QC record. Use the Suspicious Log while you write the program.

The Suspicious Log after a program with these problems: the same four NOTEs the macro counts, each with its program line.

Exercise 15-4. %logcheck already looks for the NOTE Division by zero detected. Test this rule with a step that divides by zero.

Exercises

The exercises are next to the section they belong to: 15-1 and 15-2 in 15.3, 15-3 in 15.4, 15-4 in 15.5. The answers are in Answers, Parts IV and V.