@Chef
I can reproduce your results, Kubuntu 24.04 and Gimp 2.10.38 / Gimp 3.0.4 (appimage)
A 18 MB PDF - a 60 page magazine mixed text and graphics. 35 seconds vs. 1min - 40 seconds.
If you open the Gimp dashboard, Windows -> Dockable Dialogs -> Dashboard loading the PDF is not heavy on resources much the same in 2.10 or 3.0 so not that.
I thought it might be color management, but that turned on/off make no difference.
Then if the PDF contains lots of text / fonts maybe Gimp font rendering is slow. However rendering all text to vectors using Scribus makes no difference
Gimp versions ? I can go back to Gimp 3.0.0 (appimage) and no difference there.
Probably worth a report to the devs
https://gitlab.gnome.org/GNOME/gimp/-/issues