In iOS 14.4.2, open Messages.app and scroll so the unwanted message appears. As an example, imagine the message "Fran and Bob are bringing flowers today" also includes an audio file (not shown in the attached screenshots):
Press and hold your finger on the message that contains the audio file; the audio file won't play. However, a contextual menu will appear:
Tap "More," and you'll see this:
The blue checkmark indicates the individual audio file (not the entire thread) has been selected. Touch the trash icon at the lower left, and the unwanted audio file will be deleted.
This action occurs only on your iPhone; the sender will not be notified that you have deleted the message or the file.