Consider clearing or archiving folders where mail is automatically being placed.
After trying many things (like clobbering old directories in Caches, Preferences and Containers) with only minimal benefit, I had some success by clearing out some subfolders that I don't pay much attention to, where I have less-important mail refiled by server-side rules.
When those folders had around 10K messages in them, I saw mail get stuck "Downloading messages" forever, not pull in incoming Mail for hours (until restart), or simply crash occasionally. I haven't seen any of those since clearing out that clutter, knock on wood.