I've tried deleting and reinstalling AirPort Utility, but I can still manage base stations without being asked for a configuration password.
I'm thinking of selecting not to back up data from AirPort Utility, then making a backup to iCloud and wiping and restoring the device. Has anyone tried this?