TPC

Motivation

The pressure of ever-increasing patient demand and budget restrictions make hospital bed management a daily challenge for clinical staff. Most critical is the efficient allocation of resource-heavy Intensive Care Unit (ICU) beds to the patients who need life support. Central to solving this problem is knowing for how long the current set of ICU patients are likely to stay in the unit. In this work, we propose a new deep learning model based on the combination of temporal convolution and pointwise (1x1) convolution, to solve the length of stay prediction task on the eICU and MIMIC-IV critical care datasets. The model – which we refer to as Temporal Pointwise Convolution (TPC) – is specifically designed to mitigate common challenges with Electronic Health Records, such as skewness, irregular sampling and missing data. In doing so, we have achieved significant performance benefits of 18-68% (metric and dataset dependent) over the commonly used Long-Short Term Memory (LSTM) network, and the multi-head self-attention network known as the Transformer. By adding mortality prediction as a side-task, we can improve performance further still, resulting in a mean absolute deviation of 1.55 days (eICU) and 2.28 days (MIMIC-IV) on predicting remaining length of stay.

Headline Results

Length of Stay Prediction

We report on the following metrics:

  • Mean absolute deviation (MAD)
  • Mean absolute percentage error (MAPE)
  • Mean squared error (MSE)
  • Mean squared log error (MSLE)
  • Coefficient of determination (R2)
  • Cohen Kappa Score (Harutyunyan et al. 2019)

For the first four metrics, lower is better. For the last two, higher is better.

eICU

ModelMADMAPEMSEMSLER2Kappa
Mean*3.21395.729.52.870.000.00
Median*2.76184.432.62.15-0.110.00
LSTM2.39±0.00118.2±1.126.9±0.11.47±0.010.09±0.000.28±0.00
CW LSTM2.37±0.00114.5±0.426.6±0.11.43±0.000.10±0.000.30±0.00
Transformer2.36±0.00114.1±0.626.7±0.11.43±0.000.09±0.000.30±0.00
TPC1.78±0.0263.5±4.321.7±0.50.70±0.030.27±0.020.58±0.01

Our model (TPC) significantly outperforms all baselines by large margins. *The mean and median “models” always predict 3.47 and 1.67 days respectively (the mean and median of the training set).

MIMIC-IV

Please note that this is not the same cohort as used in Harutyunyan et al. 2019. They use the older MIMIC-III database and I have developed my own preprocessing pipeline to closely match that of eICU.

ModelMADMAPEMSEMSLER2Kappa
Mean*5.24474.977.72.800.000.00
Median*4.60216.886.82.09-0.120.00
LSTM3.68±0.02107.2±3.165.7±0.71.26±0.010.15±0.010.43±0.01
CW LSTM3.68±0.02107.0±1.866.4±0.61.23±0.010.15±0.010.43±0.00
Transformer3.62±0.02113.8±1.863.4±0.51.21±0.010.18±0.010.45±0.00
TPC2.39±0.0347.6±1.446.3±1.30.39±0.020.40±0.020.78±0.01

*The mean and median “models” always predict 5.70 and 2.70 days respectively (the mean and median of the training set).

Mortality Prediction

We report on the following metrics:

  • Area under the receiver operating characteristic curve (AUROC)
  • Area under the precision recall curve (AUPRC)

For both metrics, higher is better.

eICU

ModelAUROCAUPRC
LSTM0.849±0.0020.407±0.012
CW LSTM0.855±0.0010.464±0.004
Transformer0.851±0.0020.454±0.005
TPC0.864±0.0010.508±0.005

MIMIC-IV

ModelAUROCAUPRC
LSTM0.895±0.0010.657±0.003
CW LSTM0.897±0.0020.650±0.005
Transformer0.890±0.0020.641±0.008
TPC0.905±0.0010.691±0.006

Multitask Prediction

These are the results when the model is trained to solve length of stay and mortality at the same time.

eICU

ModelAUROCAUPRCMADMAPEMSEMSLER2Kappa
LSTM0.852±0.0030.436±0.0072.40±0.01116.5±0.827.2±0.21.47±0.010.08±0.010.28±0.01
CW LSTM0.865±0.0020.490±0.0072.37±0.00115.0±0.726.8±0.11.44±0.000.09±0.000.30±0.00
Transformer0.858±0.0010.475±0.0042.36±0.00114.2±0.726.6±0.11.43±0.000.10±0.000.30±0.00
TPC0.865±0.0020.523±0.0061.55±0.0146.4±2.618.7±0.20.40±0.020.37±0.010.70±0.00

MIMIC-IV

ModelAUROCAUPRCMADMAPEMSEMSLER2Kappa
LSTM0.896±0.0020.659±0.0043.66±0.01106.8±2.765.3±0.61.25±0.010.16±0.010.44±0.00
CW LSTM0.899±0.0020.654±0.0033.69±0.02107.2±1.666.3±0.61.23±0.010.15±0.010.44±0.00
Transformer0.898±0.0010.656±0.0053.61±0.01112.3±2.063.3±0.31.20±0.010.19±0.000.45±0.00
TPC0.918±0.0020.713±0.0072.28±0.0732.4±1.242.0±1.20.19±0.000.46±0.020.85±0.00

Pre-processing Instructions

Note that soon I am planning to also implement a version of the TPC model that can simply be plugged into the Harutyunyan et al. pipeline.

Clone the GitHub repository.

eICU

  1. To run the sql files you must have the eICU database set up.

  2. Follow the instructions to ensure the correct connection configuration.

  3. Replace the eICU_path in paths.json to a convenient location in your computer, and do the same for eICU_preprocessing/create_all_tables.sql using find and replace for '/Users/emmarocheteau/PycharmProjects/TPC-LoS-prediction/eICU_data/'. Leave the extra ‘/’ at the end.

  4. In your terminal, navigate to the project directory, then type the following commands:

    psql 'dbname=eicu user=eicu options=--search_path=eicu'
    

    Inside the psql console:

    \i eICU_preprocessing/create_all_tables.sql
    

    This step might take a couple of hours.

    To quit the psql console:

    \q
    
  5. Then run the pre-processing scripts in your terminal. This will need to run overnight:

    python3 -m eICU_preprocessing.run_all_preprocessing
    

MIMIC-IV

  1. To run the sql files you must have the MIMIC-IV database set up.

  2. The official recommended way to access MIMIC-IV is via BigQuery. Personally I did not find it easy to store the necessary views and there is a 1GB size limit on the data you can download in the free tier, which is less than I am using here (the largest file to extract is timeseries.csv which is 4.49GB). However if you do wish to use BigQuery, note that you will have to make minor modifications to the code e.g. you would need to replace a reference to the table patients with physionet-data.mimic_core.patients.

    Alternatively, you can follow instructions to set up the full database. The instructions for the previous version of MIMIC - MIMIC-III are here for unix systems or here for windows. You will need to change mimiciii schema to mimiciv and use the files here in place of the files here (referenced in the instructions). Additionally you may find this resource helpful, which is still in the process of being updated (as of November 2020).

  3. Once you have a database connection, replace the MIMIC_path in paths.json to a convenient location in your computer, and do the same for MIMIC_preprocessing/create_all_tables.sql using find and replace for '/Users/emmarocheteau/PycharmProjects/TPC-LoS-prediction/MIMIC_data/'. Leave the extra ‘/’ at the end.

  4. If you have set up the database on your local computer, you can navigate to the project directory in your terminal, then type the following commands:

    psql 'dbname=mimic user=mimicuser options=--search_path=mimiciv'
    

    Inside the psql console:

    \i MIMIC_preprocessing/create_all_tables.sql
    

    This step might take a couple of hours.

    To quit the psql console:

    \q
    
  5. Then run the pre-processing scripts in your terminal. This will need to run overnight:

    python3 -m MIMIC_preprocessing.run_all_preprocessing
    

Running the models

  1. Once you have run the pre-processing steps you can run all the models in your terminal. Set the working directory to the TPC-LoS-prediction, and run the following:

    python3 -m models.run_tpc
    

    Note that your experiment can be customised by using command line arguments e.g.

    python3 -m models.run_tpc --dataset eICU --task LoS --model_type tpc --n_layers 4 --kernel_size 3 --no_temp_kernels 10 --point_size 10 --last_linear_size 20 --diagnosis_size 20 --batch_size 64 --learning_rate 0.001 --main_dropout_rate 0.3 --temp_dropout_rate 0.1 
    

    Each experiment you run will create a directory within models/experiments. The naming of the directory is based on the date and time that you ran the experiment (to ensure that there are no name clashes). The experiments are saved in the standard trixi format.

  2. The hyperparameter searches can be replicated by running:

    python3 -m models.hyperparameter_scripts.eICU.tpc
    

    Trixi provides a useful way to visualise effects of the hyperparameters (after running the following command, navigate to http://localhost:8080 in your browser):

    python3 -m trixi.browser --port 8080 models/experiments/hyperparameters/eICU/TPC
    

    The final experiments for the paper are found in models/final_experiment_scripts e.g.:

    python3 -m models.final_experiment_scripts.eICU.LoS.tpc
    

References

Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask Learning and Benchmarking with Clinical Time Series Data. Scientific Data, 6(96), 2019.

Emma Rocheteau
Emma Rocheteau
ML for Healthcare PhD Student

My research interests include deep learning for patient outcome prediction, especially time series methods and graph neural networks.

Related