MELON: Multimodal Mixture-of-Experts with Spectral-Temporal Fusion for Long-Term Mobility Estimation in Critical Care

📝 Paper was accepted and will be presented at MICCAI 2025 [SPOTLIGHT].

---

📘 Overview

MELON is a deep learning framework designed to estimate patient long-term (12 hours) mobility in the ICU using accelerometer data. It is a dual-branch architecture combining Time-MoE and ResNet for sequential data and image features together for spectral-temporal fusion.

This repository includes:

• Data preprocessing pipelines for raw accelerometer data preparation.
• Training and evaluation scripts for MELON.

---

⚙️ Requirements

Software

• Python ≥ 3.10
• Package Manager: pip or conda
• Key Python Libraries:
  • pandas
  • numpy
  • scikit-learn
  • scipy
  • torch (PyTorch)
  • transformers
  • shap
  • seaborn

Install all dependencies with:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt

Note: For GPU support with PyTorch, refer to the official installation guide.

Hardware

• CPU: Multi-core processor
• RAM: ≥ 8GB
• GPU: NVIDIA GPU with CUDA support (recommended for training deep learning models)

---

🏥 Data Sources

This project utilizes accelerometer data collected from nine specialized ICUs (Cardiology, Cardiac, Medical, Neuromedicine, Neuro-vascular, Thoracic & Lung Transplant, Trauma, and Surgery) in University of Florida between 2019 and 2024:

• ICU-Multimodal: A multi-center ICU database with high granularity data for over 400 admissions. Access requires credentialed approval.

  • Please reach out to Dr. Parisa Rashidi for more details.

---

🚀 Getting Started

1. Data Preparation and Pretrained MoE Preparation

Before using this repo, please request the following files (dataset and pretrained weights) from Dr. Parisa Rashidi:

Item	Description	Destination Path
raw_data.zip	Raw accelerometer data	data/
mobility_labels.csv	Mobility label annotations	data/
pretrained_moe.pth	Pretrained Mixture-of-Experts weights	melon/model/

After obtaining the files, extract or place them into the respective folders as shown above.

Note:

• Adjust paths and parameters as needed in the script.
• You can use your own dataset by modifying & running the Python script in ./data/preprocess.py.

2. Model Training

Navigate to the root path of the project and run the training script:

sh scripts/run_train.sh

Note: Adjust parameters as needed in the script. You may test different accelerometer types ([from arm & from ankle]) and different tasks ([mobility & activity]).

Training duration is approximately 5 minutes on an NVIDIA A100 GPU.

---

📊 Results & Performance

MELON demonstrates high performance in predicting patient mobility and activity, with AUROC scores comparable to state-of-the-art models.

---

📄 License

This project is licensed under the GNU General Public License v3.0.

---

📚 Citation

If you use this work in your research, please cite:

@article{zhang2025melon,
  title={MELON: Multimodal Mixture-of-Experts with Spectral-Temporal Fusion for Long-Term Mobility Estimation in Critical Care},
  author={Zhang, Jiaqing and Contreras, Miguel and Sena, Jessica and Davidson, Andrea and Ren, Yuanfang and Guan, Ziyuan and Ozrazgat-Baslanti, Tezcan and Loftus, Tyler J and Nerella, Subhash and Bihorac, Azra and others},
  journal={arXiv preprint arXiv:2503.11695},
  year={2025}
}

---

📬 Contact

• Dr. Parisa Rashidi: parisa.rashidi@bme.ufl.edu

---