Develop like a senior

A senior programmer does not know more keys than you. A senior has a routine: look at the data before you write code, check a number in a second way, and look inside the run instead of guessing. Each workflow below uses the tools of chapters 20 to 25 together, on the pilot data.

Workflow	The question	Tools
1	What is in this domain?	Libraries, data viewer, SAS Pry terminal
2	Why is my count too high?	data viewer, `dups`, Run Step
3	Does my macro do what it says?	keyboard macros, tests, macro debugger, coverage
4	Why do production and reference differ?	PROC COMPARE, SAS Pry terminal, DATA step debugger
5	Does every program stand on its own?	Run in Batch, autoexec, Problems

26.1 Explore a new SDTM domain in 5 minutes

Your new task is about concomitant medications, and you have never seen sdtm.cm. Spend five minutes with the data before you write the program.

Read the variables. In Libraries, open SDTM and click ▸ next to CM.

What you should see: 21 variables with type and label: CMTRT Reported Name of Drug, Med, or Therapy, CMDECOD Standardized Medication Name, CMSTDTC, and more.
Open the data. Double-click CM.

What you should see: 7510 rows.
Count the values of the coded name. Click a cell in CMDECOD and press f.

What you should see: UNCODED 6085 times, then ACETYLSALICYLIC ACID 380. Most records are not coded.
Look at the reported names. Click a cell in CMTRT and press f.

What you should see: MULTIVITAMIN 470, MULTIVITAMINS 253 and further down MULTIVIT 13: one medication, written three ways. ACCUPRIL 19 and ACUPRIL 13: a typo.
Profile every column at once. Run DataStep Studio: Open Pry Terminal. Type vt sdtm.cm, then profile.

What you should see: each column with N, missing, distinct values, min and max. CMSTDTC has 21 missing values.
Count subjects, not records. Type sql SELECT COUNT(DISTINCT usubjid) AS n FROM sdtm.cm.

What you should see: 229 subjects. Then sql SELECT COUNT(DISTINCT usubjid) AS n FROM sdtm.cm WHERE cmdecod = "UNCODED": 221.

Write down what you found, before you write the program:

sdtm.cm: 7510 records, 229 subjects, 21 variables
- 6085 records (221 subjects) are UNCODED: a table by CMDECOD is not possible as it is
- CMTRT spellings differ: MULTIVITAMIN / MULTIVITAMINS / MULTIVIT, ACCUPRIL / ACUPRIL
- 21 records without a start date: decide the imputation before you derive ASTDT

These three lines are questions for the data manager and the statistician. You found them before anyone built a table on them. To keep a copy to play with, run this and press F12:

/* Chapter 26: a first look at a domain you do not know yet */
DATA cm;
  SET sdtm.cm;
RUN;

26.2 Why is my count 25 and not 16?

The statistician checks your table of severe treatment-emergent AEs and writes: “Low dose: the number of subjects should be 16, not 25.” Here is the program:

/* Chapter 26: a count that is too high */
PROC SQL;
  CREATE TABLE sev_count AS
    SELECT trta, COUNT(*) AS n_subj
      FROM refadm.adae
      WHERE trtemfl = "Y" AND aesev = "SEVERE"
      GROUP BY trta;
QUIT;

Run it and look. Press F4, then F12.

What you should see: Placebo 6, Xanomeline High Dose 10, Xanomeline Low Dose 25. The SAS Log and the Suspicious Log are clean. A wrong number does not always leave a NOTE.
Count the same thing a second way. Open the SAS Pry terminal and type:
```
vt refadm.adae
where trtemfl = "Y" AND aesev = "SEVERE" AND trta = "Xanomeline Low Dose"
count
```
What you should see: 25 rows. The program counted rows.
Find the subjects behind the rows. Type dups usubjid.

What you should see: 7 subjects have more than one record: 01-708-1019 and 01-708-1428 have 3 each, five more have 2. These 7 subjects have 16 records. 9 more subjects have one record each. Total: 25 records, 16 subjects.
Fix and run only that step. Change COUNT(*) to COUNT(DISTINCT usubjid), put the cursor in the step and choose Run Step.

/* Chapter 26: the same count, one subject counted once */
PROC SQL;
  CREATE TABLE sev_count AS
    SELECT trta, COUNT(DISTINCT usubjid) AS n_subj
      FROM refadm.adae
      WHERE trtemfl = "Y" AND aesev = "SEVERE"
      GROUP BY trta;
QUIT;

What you should see: Placebo 5, Xanomeline High Dose 8, Xanomeline Low Dose 16.

26.3 Build a utility macro test-first

People call a utility macro hundreds of times and never read its code. So write what it must do as tests before you write it. The macro here is %n_obs: the number of observations of a data set, with an optional WHERE, as a number for %IF or a title.

Write the tests first. Each test calls the macro and says what it must return. The expected values are facts you know: ADSL has 254 subjects, ADAE has 1126 TEAE records, 41 of them severe.

/* EXPECT: ERROR [N_OBS] */
/* Chapter 26: the tests of %n_obs, written before the macro */
%MACRO assert_eq(actual, expected, desc);
  %IF %SUPERQ(actual) = %SUPERQ(expected) %THEN %PUT NOTE: [TEST] pass: &desc.;
  %ELSE %PUT %STR(ERR)OR: [TEST] FAIL: &desc.: got &actual., expected &expected.;
%MEND assert_eq;

%assert_eq(%n_obs(data=refadm.adsl), 254, all subjects of ADSL)
%assert_eq(%n_obs(data=refadm.adae, where=%STR(trtemfl = "Y")), 1126, treatment-emergent AE records)
%assert_eq(%n_obs(data=refadm.adae, where=%STR(trtemfl = "Y" AND aesev = "SEVERE")), 41, severe TEAE records)
%assert_eq(%n_obs(data=refadm.adae, where=%STR(aesev = "NONE")), 0, a WHERE that keeps nothing)
%assert_eq(%n_obs(data=work.nothere), -1, a data set that does not exist)

Run the tests: red. Press F4.

What you should see: WARNING: Apparent invocation of macro N_OBS not resolved. and five lines ERROR: [TEST] FAIL: .... Good: the tests can fail. A test that cannot fail tests nothing.

Write the macro. In a new file, type headerm and press Tab for the header, then write the code. %required, %error and %exist + Tab give you the checks.

/*******************************************************************************
| Macro Name    : n_obs
| Macro Version : v1.00
| Purpose       : Number of observations of a data set, with an optional WHERE.
|                 A function-style macro: it returns a number, so it can be used
|                 in %IF, %LET and in titles.
|
| Parameters
|   data  = input data set, required             Example: refadm.adae
|   where = filter, default 1 (all records)       Example: %STR(trtemfl = "Y")
|
| Returns -1 and writes an ERROR when the data set cannot be opened.
|
| Usage
|   %PUT NOTE: %n_obs(data=refadm.adae, where=%STR(trtemfl = "Y")) TEAE records;
|
| Change log
|   v1.00  first version, written test first (chapter 26)
*******************************************************************************/
%MACRO n_obs(data=, where=1);
  %LOCAL dsid n rc;
  %LET n = -1;
  %IF %LENGTH(&data.) = 0 %THEN %DO;
    %PUT %STR(ERR)OR: [N_OBS] DATA= is required.;
    %GOTO finish;
  %END;
  %IF NOT %SYSFUNC(EXIST(&data.)) %THEN %DO;
    %PUT %STR(ERR)OR: [N_OBS] Data set &data. does not exist.;
    %GOTO finish;
  %END;
  %LET dsid = %SYSFUNC(OPEN(&data.(WHERE=(%UNQUOTE(&where.)))));
  %IF &dsid. = 0 %THEN %DO;
    %PUT %STR(ERR)OR: [N_OBS] %SYSFUNC(SYSMSG());
    %GOTO finish;
  %END;
  %LET n  = %SYSFUNC(ATTRN(&dsid., NLOBSF));
  %LET rc = %SYSFUNC(CLOSE(&dsid.));
  %finish:
  &n.
%MEND n_obs;

Compile it, then run the tests: green. Run the macro file with F4, then the tests again.

What you should see: five lines NOTE: [TEST] pass: ..., and one ERROR: [N_OBS] Data set work.nothere does not exist., which the last test asks for.
Ask the coverage what you did not test. Write the call %PUT NOTE: %n_obs(data=refadm.adae, where=%STR(aesev = "NONE")); below the macro, put the cursor on %n_obs and click Debug Macro (chapter 25). Click Start Debug, then F5 to the end.

What you should see: three checks are marked as not reached: %IF %LENGTH(&data.) = 0, %IF NOT %SYSFUNC(EXIST(&data.)) and %IF &dsid. = 0. The test with work.nothere covers the EXIST check. The other two are gaps: no test calls the macro without DATA=, and no test has a WHERE that SAS cannot use.
Add the missing test. A WHERE on a variable that does not exist:
```
%assert_eq(%n_obs(data=refadm.adae, where=%STR(nosuch = 1)), -1, a WHERE on a missing variable)
```
What you should see: ERROR: [N_OBS] ERROR: Variable nosuch is not on file REFADM.ADAE. and NOTE: [TEST] pass: a WHERE on a missing variable.
Put it where everyone finds it. Save 26_4_n_obs.sas as n_obs.sas in a macros folder of your project. Now Cmd/Ctrl+click on any %n_obs call opens it, and %n_o + Enter writes a call with all parameters.

This is the method of chapter 19: the tests are part of the macro. When you change it next year, the tests tell you in seconds that nothing broke.

26.4 Chase a PROC COMPARE difference

In chapter 15, your ADSL and the CDISC reference differ in TRTEDT for 6 subjects. A senior does not change the rule before they know why.

Run production and the compare. Run 15_1_production.sas, then 15_3_compare.sas.

What you should see: in SAS Output, the third compare lists six subjects. 01-705-1018 has 12JUL2013 in the reference and . in yours, and 01-704-1233 has 14JUL2013 and 04APR2013.
Look at the source records. Open the SAS Pry terminal:
```
vt sdtm.ex
where usubjid = "01-705-1018"
show
```
What you should see: one record: EXSTDTC 2013-07-05, EXENDTC empty. Then where usubjid = "01-704-1233" and show: two records, the first from 2013-03-21 to 2013-04-04, the second from 2013-04-05 without an end date.
Watch your rule on that subject. Put the cursor in the DATA step prod_trt of 15_1_production.sas and click Debug DATA Step. Set a breakpoint on the line IF NOT MISSING(exendtc) ... with the condition usubjid = "01-704-1233" and press F5 twice.

What you should see: on the second record exendtc is empty, so trtedt keeps 04APR2013 from the first record. Your rule, “the latest EX end date”, does exactly what it says.
Write the finding down. The code does what the rule says. The rule disagrees with the reference for open EX records. Ask the statistician which rule the specification wants, then change the code. Chapter 9 tells the whole TRTEDT story.

You can also compare from the terminal: vt prod, then compare refadm.adsl by usubjid, runs PROC COMPARE of your data set against the reference without a program.

26.5 Run a chapter in batch

The programs of this book run in one session, in file order, so a program can use the WORK data of an earlier one. A real study is different: each program must run on its own, in a fresh session, in the night run. Run in Batch shows you which programs cannot.

Give batch sessions your librefs. Open the settings, search for datastepStudio.autoexec, and add the two LIBNAME lines of 00_libs.sas after OPTIONS NOTES;.
Run the ADaM programs in batch. In the Explorer, select 09_1_adsl.sas, 09_2_adsl_check.sas and 11_1_adae.sas, right-click → Run in Batch. Keep working while they run.

What you should see: when it is done, a message says how it went. 11_1_adae.sas has an error. The second button opens its log: ERROR: File WORK.ADSL.DATA does not exist.
Read the summary. Show Summary lists each program and its result. The .log and .lst files are next to each program.
Make the program stand on its own. 11_1_adae.sas needs ADSL, and in a fresh session WORK is empty. In a study, the ADSL program writes adam.adsl to a permanent library and the ADAE program reads it from there. Then the batch order, 09_ before 11_, is the only thing that connects them.

A folder runs its own .sas files, not those of its subfolders. Cancel stops after the running program.

Exercises

Exercise 26-1. Do workflow 1 for sdtm.lb. Which LBTESTCD values are there, and which units?

Exercise 26-2. In workflow 2, how many records and subjects are behind the High Dose count?

Exercise 26-3. Add a test to 26_5_n_obs_tests.sas for a call without DATA=. What must %n_obs return?

Exercise 26-4. Try %n_obs(data=refadm.adsl(WHERE=(saffl = "Y"))). What happens, and why? Use %pry in the macro to see the text that goes into OPEN.

Answers: Chapter 26 answers.

Next: the cheat sheet.