Structure of the published data

Overview

The overall data is available in two states: as raw data and/or as pre-processed data. Additionally there are three reference tables for variable lookup.

Reference tables

• variable reference (*hirid_variable_reference.csv*) - reference table for variables (for raw stage)
• ordinal variable reference (*ordinal_vars_ref.csv*) - reference table for categorical/ordinal variables for string value lookup
• pre-processed variable reference (*hirid_variable_reference_preprocessed.csv*) - reference table for variables (for merged and imputed stage)

Raw data

The raw data was only processed if this was necessary for patient de-identification and otherwise left unchanged compared to the original source. The raw data contains the complete set of available variables (681 variables). It consists of the following tables:

• observations
• pharma records
• general data

Pre-processed data

The pre-processed data consists of intermediary pipeline stages from our original Nature medicine publication. Source variables representing the same clinical concepts were merged into one meta-variable per concept. The data contains the 18 most predictive meta-variables only, as defined in our publication. Two different stages of the pipeline are available

• Merged stage source variables are merged into meta-variables by clinical concepts e.g. non-opioid-analgesics. The time grid is left unchanged and is sparse.
• Imputed stage the data from the merged stage is down sampled to a five-minute time grid. The time grid is filled with imputed values. The imputation strategy is complex and is discussed in the original publication.

The code used to generate these stages can be found in this GitHub repo under the preprocessing folder.

Which data to use?

The pre-processed data is intended mainly as a quick way to jump-start a project or for use in a proof of concept. We recommend using the source data whenever possible for regular projects. It is the most flexible form and contains the complete set of variables in the original time resolution.

Data formats

Data is available in two formats: CSV for wide compatibility and Apache Parquet for convenience and performance. Parquet is a strongly typed, binary format that is supported by many major data processing tools such as pandas, spark, R, matlab, etc.

Since the data sets are fairly large, they are split into partitions, such that they can be processed in parallel in a straightforward way. The lookup table mapping patient id to partition id is provided in the file named {data_set}_index.csv along with the data. The partitions are aligned between the different data sets and tables, such that the data of a patient can always be found in the partition with the same id. Note however, that a patient may not occur in all data sets, e.g. a patient might be missing in the preprocessed data, because a patient didn't meet the demographic criteria to be included in the study.

Data schemata