Unfortunately, most streaming players I'm currently using, except YouTube, have this exact same problem. Sometimes they simply don't play, even though they should have been loaded far enough. But in other times, they work perfectly fine. So, it's surely not that the video is defective.
Another thing which also happens regularly on these streaming players is, when I pause the video I always have to fear that it doesn't play anymore, even though it should have been loaded far enough, and hence I have to reload the whole site. Jumping to different places on the progress bar sometimes changes the still image, but it doesn't play nonetheless.
And sometimes the video plays again after a moment of waiting (it can go up to a few minutes) which is still strange since it has already loaded quite far and just by pausing the video shouldn't have made it load again.
Since none of these problems occur on YouTube, does that platform use a different way to play its videos or does Safari play a part in it? And is there a way to fix that problem?