I had a similar problem on 2001 V70 T5 and solved by replacing the ETM with Exemodex. You should have the the improved later ETM I think, but maybe not. Search ETM here for lots of info.
volvolugnut

Theres harness that runs behind the starter which can give hard hot starts, moving it away from the starter helps, its known to pick up RF interference. It might be the crank position sensor wire.

i had this, mine turned out to be the ignition key imobilizer ring, it cranked and cranked after a short drive on hot days. However it can also do it if you have low compression.

The 2004 is known for having a CEM heat soak issue in warm climates - there ar threads on that and adding a heat sink as a solution

Try this, remove the ignition immobiliser from the ignition and hit it with a heat gun to simulate a hot day, place back in position and try starting. Have you ever seen the message "start inhibited please try again" after many months of starting issues my car only displayed this once but ...

Thank you all for the replies, I finally had some time to try some new fixes. I believe I fixed my rough idle problem today by replacing the fuel pressure sensor and removing the blowoff valve which had a bad seal. Logging fuel pressure with VIDA showed some large spikes in fuel pressure randomly a...

99% it's the starter, as others already said.