Looks like it wasn't downloading the thing, I checked in activity monitor after deleting the folder. It copies it from somewhere else, most likely
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/
Xcode simulator constantly download something
I removed all the listed items from Xcode's Window Menu → devices and simulators, which listed iOS 12 as their OS version.
Went to iOS settings → Developer and cleared paired devices.
Connected iPhone, unlocked it and chose "Don't Trust" upon prompt.
No more unnecessary folders.
Since it causes problem for iTunes backup and network sharing, Go to `/Users/ankitkumar/Library/Developer/Xcode/iOS DeviceSupport/12.4` right click on it > get info > Check locked.