Introduction to healthdb

What it does

Motivation

In health research and surveillance, identifying diseases or events from administrative databases is often the initial step. However, crafting case-finding algorithms is a complex task. Existing algorithms, often written in SAS by experienced analysts, can be complex and difficult to decipher for the growing number of analysts trained primarily in R.

These algorithms may also affect performance if they depend on Data Step in SAS, due to a lack of translation between Data Step and SQL. This can result in SAS downloading data from a remote database to a local machine, leading to poor performance when handling large, population-based databases.

The ‘healthdb’ R package was created to address these challenges. It minimizes the need to download data and offers an easy-to-use interface for working with healthcare databases. It also includes capabilities not supported by ‘SQL’, such as matching strings by ‘stringr’ style regular expressions, and can compute comorbidity scores (compute_comorbidity()) directly on a database server. This vignette will present an example of common use cases.

Installation

Simply run:

install.packages("healthdb")

We will need the following packages for this demo.

library(dplyr)
#> Warning: package 'dplyr' was built under R version 4.3.2
library(dbplyr)
#> Warning: package 'dbplyr' was built under R version 4.3.3
library(lubridate)
#> Warning: package 'lubridate' was built under R version 4.3.2
library(glue)
#> Warning: package 'glue' was built under R version 4.3.3
library(purrr)
#> Warning: package 'purrr' was built under R version 4.3.2
library(healthdb)

Intended use case

Consider the case definition of substance use disorder (SUD) from British Columbia Centre for Disease Control’s Chronic Disease Dashboard,

One or more hospitalization with a substance use disorder diagnostic code, OR Two or more physician visits with a substance use disorder diagnostic code within one year.

We are going to implement this definition. First, let’s make a demo data sets for the two sources:

  1. Physician claims with multiple columns of ICD-9 diagnostic codes

    # make_test_dat() makes either a toy data.frame or database table in memory with known number of rows that satisfy the query we will show later
claim_db <- make_test_dat(vals_kept = c("303", "304", "305", "291", "292", glue("30{30:59}"), glue("29{10:29}"), noise_val = c("999", "111")), type = "database")

# this is a database table
# note that in-memory SQLite database stores dates as numbers
claim_db %>% head()
#> # Source:   SQL [6 x 6]
#> # Database: sqlite 3.45.2 [:memory:]
#>     uid clnt_id dates diagx diagx_1 diagx_2
#>   <int>   <int> <dbl> <chr> <chr>   <chr>  
#> 1    50       1 17853 3037  2917    3049   
#> 2    38       1 18118 3059  2911    <NA>   
#> 3    14       2 16734 2920  2926    999    
#> 4    98       3 16809 999   999     <NA>   
#> 5    90       3 18003 999   <NA>    <NA>   
#> 6    12       3 18163 3056  291     999

  2. Hospitalization with ICD-10 codes

    hosp_df <- make_test_dat(vals_kept = c(glue("F{10:19}"), glue("F{100:199}"), noise_val = "999"), type = "data.frame")

# this is a local data.frame/tibble
hosp_df %>% head()
#>   uid clnt_id      dates diagx diagx_1 diagx_2
#> 1  56       1 2018-05-22   999    <NA>    <NA>
#> 2  73       1 2019-01-25   999     999    <NA>
#> 3  10       1 2019-09-01  F161    F154     999
#> 4  94       2 2017-01-21   999     999     999
#> 5  12       2 2017-03-27  F180    F123    <NA>
#> 6  98       2 2017-04-30   999     999     999

# convert Date to numeric to be consistent with claim_db
hosp_df <- hosp_df %>% 
  mutate(dates = julian(dates))

Interactive functions

Let’s focus on the physician claims. Extracting clients with at least two records within a year is not difficult, and involves only a few steps. The codes could look like the following using dplyr, however, it does not work because: 1. SQL does not support multiple patterns in one LIKE operation, 2. dbply currently have issue with translating n_distinct.

## not run
claim_db %>%
  # identify the target codes
  filter(if_any(starts_with("diagx"), ~ str_like(., c("291%", "292%", "303%", "304%", "305%")))) %>%
  # each clnt has at least 2 records on different dates
  group_by(clnt_id) %>%
  # the n_distinct step is mainly for reducing computation in the next step
  filter(n_distinct(dates) >= 2) %>%
  # any two dates within one year?
  filter((max(dates) - min(dates)) <= 365)
## end

Here’s how you could use healthdb to achieve these steps:

  1. Identify rows contains the target codes. Use ?identify_row to see a list of supported matching types.

    result1 <- claim_db %>%
  identify_row(
vars = starts_with("diagx"),
match = "start",
vals = c(291:292, 303:305)
  )
#> ℹ Identify records with condition(s):
#> • where at least one of the diagx, diagx_1, diagx_2 column(s) in each record
#> • contains a value satisfied SQL LIKE pattern: 291% OR 292% OR 303% OR 304% OR 305%
#> ℹ To see the final query generated by 'dbplyr', use dplyr::show_query() on the output.
#> To extract the SQL string, use dbplyr::remote_query().

  2. Bonus: remove clients with exclusion codes

    This step is not in the substance use disorder definition, but other disease definitions often require exclusion of some ICDs that contradicts the ones of interest. Let’s say we want to remove clients with code “111” here.

    We first identify “111” from the source, then exclude clients in the output from the previous step’s result. exclude() take either a data set (via the excl argument) or expression (condition argument) as input. For the former, it performs an anti join matching on the by argument (see dplyr::join_by()). For the latter, it is the opposite of filter, i.e., filter(!(some_expression)).

    result2 <- result1 %>%
  exclude(
excl = identify_row(claim_db, starts_with("diagx"), "in", "111"),
by = "clnt_id"
  )
#> ℹ Identify records with condition(s):
#> • where at least one of the diagx, diagx_1, diagx_2 column(s) in each record
#> • contains a value exactly matched values in set: "111"
#> ℹ Exclude records in `data` through anti_join with `excl` matching on (by argument): "clnt_id"

  3. Restrict the number of records per client

    result3 <- result2 %>% restrict_n(
  clnt_id = clnt_id,
  n_per_clnt = 2,
  count_by = dates,
  # here we use filter mode to remove records that failed the restriction
  mode = "filter"
)
#> ℹ Apply restriction that each client must have at least 2 records with distinct
#> dates. Clients/groups which did not met the condition were excluded.

  4. Restrict the temporal pattern of diagnoses

    restrict_date() supports more complicated patterns like having n diagnoses at least i days apart within j years. Note that when SQL interpret order of dates, the result could be not deterministic if there were duplicate dates within client. Therefore, a unique row id (uid) has to be supplied to get consistent result.

    result4 <- result3 %>% restrict_date(
  clnt_id = clnt_id,
  date_var = dates,
  n = 2,
  within = 365,
  uid = uid,
  # here we use flag mode to flag records that met the restriction instead of removing those
  mode = "flag"
)
#> ℹ Apply restriction that each client must have 2 records that were within 365
#> days. Records that met the condition were flagged.

  5. Fetch variables from other tables by matching common keys

    Up to this point, the result is only a query and have not been downloaded. Hopefully, the data has been shrunken to a manageable size for collection.

    # Class of result4
class(result4)
#> [1] "tbl_SQLiteConnection" "tbl_dbi"              "tbl_sql"             
#> [4] "tbl_lazy"             "tbl"

# execute query and download the result
result_df <- result4 %>% collect()

# Number of rows in source
nrow(claim_db %>% collect())
#> [1] 100

# Number of rows in the current result
nrow(result_df)
#> [1] 27

    Our data now only contains diagnoses which are probably not enough for further analyses. Let’s say we want to gather client demographics such as age and sex from other sources. This certainly can be done with multiple dplyr::left_join() calls. Here we provide the fetch_var() function to make the codes more concise.

    # make two look up tables
age_tab <- data.frame(
  clnt_id = 1:50,
  age = sample(1:90, 50),
  sex = sample(c("F", "M"), 50, replace = TRUE)
)
address_tab <- data.frame(
  clnt_id = rep(1:50, 5), year = rep(2016:2020, each = 50),
  area_code = sample(0:200, 50, replace = TRUE)
)

# get year from dates for matching

result_df <- result_df %>% mutate(year = lubridate::year(as.Date(dates, origin = "1970-01-01")))

# note that keys must be present in all tables
result_df %>%
  fetch_var(
keys = c(clnt_id, year),
linkage = list(
  # the formula means from_table ~ get_variable
  # |clnt_id means matching on clnt_id only
  age_tab ~ c(age, sex) | clnt_id,
  address_tab ~ area_code
)
  ) %>%
  select(uid, clnt_id, dates, age, sex, area_code) %>% 
  head()
#> # A tibble: 6 × 6
#>     uid clnt_id dates   age sex   area_code
#>   <int>   <int> <dbl> <int> <chr>     <int>
#> 1    50       1 17853    47 M            44
#> 2    38       1 18118    47 M            44
#> 3    16       8 16841    80 F           193
#> 4    45       8 17508    80 F           193
#> 5     9      12 18444    68 M            65
#> 6    48      12 18512    68 M            65

Call-building functions

To complete the definition, we need to repeat the process shown above with hospitalization data. Some studies may use more than a handful of data sources to define their sample. We packed steps 1-4 in one function define_case(), and provide tools to perform batch execution with different data and parameters to meet those needs.

# build the full definition of SUD
sud_def <- build_def(
  # name of definition
  def_lab = "SUD",
  # place holder names for sources
  src_labs = c("claim", "hosp"),
  def_fn = define_case, # you could alter it and supply your own function
  # below are argumets of define_case
  fn_args = list(
    # if length = 1, the single element will be use for every source
    vars = list(starts_with("diagx")),
    match = "start", # match ICD starts with vals
    vals = list(c(291:292, 303:305), glue("F{10:19}")),
    clnt_id = clnt_id,
    n_per_clnt = c(2, 1),
    date_var = dates,
    within = c(365, NULL),
    uid = uid,
    mode = "flag"
  )
)

sud_def
#> # A tibble: 2 × 5
#>   def_lab src_labs def_fn      fn_args          fn_call   
#>   <chr>   <chr>    <chr>       <list>           <list>    
#> 1 SUD     claim    define_case <named list [9]> <language>
#> 2 SUD     hosp     define_case <named list [9]> <language>

Let’s look inside the fn_call list column. Two calls of define_case() have been made with different parameters. The data arguments are left empty on purpose for re-usability. For example, you may want to repeat the analysis with data from different regions or study periods.

sud_def$fn_call
#> [[1]]
#> define_case(data = , vars = starts_with("diagx"), match = "start", 
#>     vals = c(291:292, 303:305), clnt_id = clnt_id, n_per_clnt = 2, 
#>     date_var = dates, within = 365, uid = uid, mode = "flag")
#> 
#> [[2]]
#> define_case(data = , vars = starts_with("diagx"), match = "start", 
#>     vals = glue("F{10:19}"), clnt_id = clnt_id, n_per_clnt = 1, 
#>     date_var = dates, within = NULL, uid = uid, mode = "flag")

Executing the definition is simply a call of execute_def(). If verbose option is not turned off by options(healthdb.verbose = FALSE), the output message will explain what has been done. You could append multiple build_def() outputs together and execute them all at once. Definition and source labels will be added to the result to identify outputs from different calls.

# execute the definition
result_list <- sud_def %>%
  execute_def(with_data = list(
    claim = claim_db,
    hosp = hosp_df
  ))
#> 
#> Actions for definition SUD using source claim_db:
#> → --------------Inclusion step--------------
#> ℹ Identify records with condition(s):
#> • where at least one of the diagx, diagx_1, diagx_2 column(s) in each record
#> • contains a value satisfied SQL LIKE pattern: 291% OR 292% OR 303% OR 304% OR 305%
#> → --------------No. rows restriction--------------
#> 
#> ℹ Apply restriction that each client must have at least 2 records with distinct dates. Records that met the condition were flagged.
#> → --------------Time span restriction--------------
#> 
#> ℹ Apply restriction that each client must have 2 records that were within 365 days. Records that met the condition were flagged.
#> → -------------- Output all records--------------
#> 
#> 
#> Actions for definition SUD using source hosp_df:
#> 
#> → --------------Inclusion step--------------
#> 
#> ℹ Identify records with condition(s):
#> • where at least one of the diagx, diagx_1, diagx_2 column(s) in each record
#> • contains a value satisfied regular expression: ^F10|^F11|^F12|^F13|^F14|^F15|^F16|^F17|^F18|^F19
#> 
#> All unique value(s) and frequency in the result (as the conditions require just one of the columns containing target values; irrelevant values may come from other vars columns): 
#>  999 F100 F102 F103 F104 F107 F108  F11 F113 F114 F116 F117 F120 F121 F123 F124 
#>    1    1    1    1    1    1    1    1    1    1    1    1    1    1    1    1 
#> F125 F126 F127 F129 F130 F131 F132 F134 F135 F136 F137 F138 F139 F140 F141 F142 
#>    1    1    1    1    1    1    1    1    1    1    1    1    1    1    1    1 
#> F143 F144 F145 F146 F148 F149  F15 F150 F151 F153 F154 F155 F157 F158 F159  F16 
#>    1    1    1    1    1    1    1    1    1    1    1    1    1    1    1    1 
#> F160 F161 F164 F165 F167 F168 F169  F17 F171 F172 F174 F175 F176 F177 F178  F18 
#>    1    1    1    1    1    1    1    1    1    1    1    1    1    1    1    1 
#> F180 F182 F184 F186 F188 F190 F194 F195 F198 F199 NA's 
#>    1    1    1    1    1    1    1    1    1    1    1
#> → -------------- Output all records--------------

Let’s check the results!

# view the results
purrr::walk(result_list, ~ head(.) %>% print())
#> # Source:   SQL [6 x 10]
#> # Database: sqlite 3.45.2 [:memory:]
#>   def   src     uid clnt_id dates diagx diagx_1 diagx_2 flag_restrict_n
#>   <chr> <chr> <int>   <int> <dbl> <chr> <chr>   <chr>             <int>
#> 1 SUD   claim    50       1 17853 3037  2917    3049                  1
#> 2 SUD   claim    38       1 18118 3059  2911    <NA>                  1
#> 3 SUD   claim    14       2 16734 2920  2926    999                   0
#> 4 SUD   claim    12       3 18163 3056  291     999                   0
#> 5 SUD   claim     4       4 16731 2918  3035    <NA>                  0
#> 6 SUD   claim    40       5 17048 3034  303     999                   0
#> # ℹ 1 more variable: flag_restrict_date <int>
#>   def  src uid clnt_id dates diagx diagx_1 diagx_2
#> 1 SUD hosp  10       1 18140  F161    F154     999
#> 2 SUD hosp  12       2 17252  F180    F123    <NA>
#> 3 SUD hosp   3       4 16693  F120    F132     999
#> 4 SUD hosp   2       7 18020  F160     F16    <NA>
#> 5 SUD hosp   1       8 17128  F182    F135     999
#> 6 SUD hosp  39       8 18142   F16    F100    <NA>

At this point, the result from the claim database (result[[1]]) has not been collected locally. You could collect it manually, do further filtering, and then combine with the result from hospitalization data in any way you want. If you just need a simple row bind, we have bind_source() with convenient naming feature.

bind_source(result_list,
  # output_name = c(names in the list elements)
  src = "src",
  uid = "uid",
  clnt_id = "clnt_id",
  flag = c("flag_restrict_date", NA),
  # force_proceed is needed to collect remote tables to local memory
  force_proceed = TRUE
)
#> # A tibble: 99 × 5
#>    src_No src     uid clnt_id  flag
#>     <int> <chr> <int>   <int> <int>
#>  1      1 claim    50       1     1
#>  2      1 claim    38       1     0
#>  3      1 claim    14       2     0
#>  4      1 claim    12       3     0
#>  5      1 claim     4       4     0
#>  6      1 claim    40       5     0
#>  7      1 claim    28       6     0
#>  8      1 claim    16       8     0
#>  9      1 claim    45       8     0
#> 10      1 claim     9      12     1
#> # ℹ 89 more rows

pool_case() goes a few steps further than row bind. It also filters records with valid flags and can summarize by client/group. Since we had to decide which variables to be summarized in advance, the output may not be flexible enough to meet your needs.

pool_case(result_list,
          def = sud_def,
          # your could skip summary with output_lvl = "raw"
          output_lvl = "clnt",
          # include records only from sources having valid records, see function documentation for more detail and other options
          include_src = "has_valid",
          force_proceed = TRUE)
#> # A tibble: 35 × 8
#>    def   clnt_id first_valid_date last_entry_date raw_in_claim raw_in_hosp
#>    <chr>   <int>            <dbl>           <dbl>        <dbl>       <dbl>
#>  1 SUD         1            17853           18140            2           1
#>  2 SUD         2            17252           17252            0           1
#>  3 SUD         4            16693           16693            0           1
#>  4 SUD         7            18020           18020            0           1
#>  5 SUD         8            17128           18142            0           2
#>  6 SUD         9            18145           18145            0           1
#>  7 SUD        10            17280           17280            0           1
#>  8 SUD        12            18198           18512            2           1
#>  9 SUD        14            18464           18464            0           1
#> 10 SUD        16            18625           18625            0           1
#> # ℹ 25 more rows
#> # ℹ 2 more variables: valid_in_claim <int>, valid_in_hosp <int>