UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcription

Reza Basiri; Ali Abedi; Chau Nguyen; Milos R. Popovic; Shehroz S. Khan

doi:10.1007/978-3-031-88220-3_16

Research

Conference paper, 2024

UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcription

Reza Basiri, Ali Abedi, Chau Nguyen, Milos R. Popovic, Shehroz S. Khan

ICPR 2024 workshops (PRHA), Lecture Notes in Computer Science (Springer), 221-232. DOI 10.1007/978-3-031-88220-3_16

In brief

UlcerGPT pairs a vision-language model (LLaVA) with a large language model (ChatGPT) to turn a photo of a diabetic foot ulcer into a written clinical description: what is in the image, where the regions of concern are, and how they should be classified. Expert clinicians rated its output on a public dataset.

Key findings

Detects, classifies and localizes regions of interest in one transcription step
Evaluated by expert clinicians for accuracy and efficiency
Aimed at telemedicine, where a specialist may never see the wound in person

Read at the publisher arXiv preprint

Themes

Keywords

Cite

Basiri, R., Abedi, A., Nguyen, C., Popovic, M. R., & Khan, S. S. (2024). UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcription. ICPR 2024 workshops (PRHA), Lecture Notes in Computer Science (Springer), 221-232. https://doi.org/10.1007/978-3-031-88220-3_16

BibTeX

@inproceedings{basiri2024ulcergpt,
  title = {{UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcription}},
  author = {Reza Basiri and Ali Abedi and Chau Nguyen and Milos R. Popovic and Shehroz S. Khan},
  booktitle = {ICPR 2024 workshops (PRHA), Lecture Notes in Computer Science (Springer)},
  year = {2024},
  pages = {221--232},
  doi = {10.1007/978-3-031-88220-3_16}
}

UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcription

In brief

Key findings

Related work

Temporal machine learning framework for diabetic foot ulcer healing trajectory prediction

Multimodal Artificial Intelligence System to Enhance the Classification and Analysis of Diabetic Foot Ulcer Healing Conditions

Applications of Generative AI in Diabetic Foot Ulcer Treatment