How Do I Fully Flush Cached Redirects From Safari?

Question

I have a device with a web-based control panel, and accidentally set it to redirect all http pages to https, even though some don't work over https. Although I've since corrected this, Safari seems to have memorised the redirect and is refusing to forget it, instead constantly attempting to redirect me to the invalid https address.

I've already closed Safari, cleared ~/Library/Caches/com.apple.Safari/ and ~/Library/Cookies/HSTS.plist but it still seems to be remembering the redirect when I re-open it.

Where else could Safari be storing this information? I can access the correct page via Firefox or Chrome, so it may not be a system wide service, or if it is it's not one that the other browsers use.

Unfortunately because the web-panel is provided by a device I don't believe I can adjust headers or setup a redirect back to the correct URL, which seem to be options offered in other similar questions, so I really need to find out where this data is being stored so I can destroy it with fire.

Answer 1

Based on quanta’s answer:

I wasn’t able to use launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist because I have System Integrity Protection enabled:

$ launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist
/System/Library/LaunchAgents/com.apple.nsurlstoraged.plist: Operation not permitted while System Integrity Protection is engaged

However I was able to work around it by doing the following:

• killall nsurlstoraged (stops your user’s nsurlstoraged process; I actually ran sudo killall nsurlstoraged, but I suspect it isn’t necessary to stop the system’s nsurlstoraged as well, since the cache is in the user Library folder)
• rm -f ~/Library/Cookies/HSTS.plist (deletes the HSTS cache)
• launchctl start /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist (restarts nsurlstoraged)

Answer 2

If you enable Develop menu in Safari preferences, you can clear cache from there (CMD+ALT+E).

Can you confirm that opening the device's control panel in Safari's Private window (or different web browser) works correctly?

Answer 3

Based on @Haravikk's answer: https://apple.stackexchange.com/a/267783/62907

Anyone have any ideas which process is responsible for the ~/Library/Cookies/HSTS.plist file?

fs_usage can help:

❯❯❯❯ sudo fs_usage | grep HSTS
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000238   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000009   nsurlstorage
16:11:03  open              /Users/quanta/Library/Cookies/HSTS.plist                                         0.016268   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000008   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000003   nsurlstorage
16:11:03  access            /Users/quanta/Library/Cookies/HSTS.plist                                         0.000011   dbfseventsd
16:11:04  lstat64           /Users/quanta/Library/Cookies/HSTS.plist                                         0.000008   fseventsd
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000006   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000002   nsurlstorage
16:11:08  open              /Users/quanta/Library/Cookies/HSTS.plist                                         0.000144   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000002   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000003   nsurlstorage
16:11:08  access            /Users/quanta/Library/Cookies/HSTS.plist                                         0.000021   dbfseventsd
16:11:09  lstat64           /Users/quanta/Library/Cookies/HSTS.plist                                         0.000042   fseventsd

So, we can:

launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist

then:

rm -f ~/Library/Cookies/HSTS.plist

and try again.

Answer 4

In Safari, Firefox and Chrome as well, all you have to do is open the developer sidebar, select the network tab, and disable caching.

In Safari that's a crossed out tube thing, the blue one next to the trash bin logo. Activate that, and the old permanent redirect should be ignored.

Greatest benefit is that you don't have to mess with files, you won't delete all HTST entries and loose the security benefits. Also it works across browsers.

Answer 5

So I've found a workaround to the problem, though this isn't a definitive answer to the actual question so I will not mark it as such until I can find more information.

It turns out that the file ~/Library/Cookies/HSTS.plist was indeed the source of the problem as I suspected, however deleting it from the affected user account doesn't work, even with Safari closed, as it is recreated after an unknown amount of time, complete with the offending entry that was forcing the invalid redirect.

So my solution was the following:

1. Make sure you have at least one other user account on your Mac (if not, create one).
2. Logout of the affected user account.
3. Login to a different user account (a guest account may not be sufficient, depending upon restrictions).
4. Find out the short-name of your affected user account; if you don't know then the best way to check is to look under System Preferences -> Users. Usually if will be the full name, lower-cased and with no spaces, so if your full name is "John Smith" then the short-name may be "johnsmith".
5. Open a window in Terminal, type su shortname replacing "shortname" with the short-name of the affected user-account. Hit enter and, when prompted, enter the password for the affected account.
6. Now type the next command rm ~/Library/Cookies/HSTS.plist and hit enter, this will delete the HSTS storage file.
7. Finally type exit, hit enter and close Terminal.

At this point you can now log back into the affected user account and the offending HSTS redirect should be gone for good.

Now, while this provides a usable workaround, I'd really like to know why deleting the HSTS.plist file from my affected account didn't work; the fact it is recreated means some background process is responsible for it, which means it should be possible to delete the file from the affected user account by simply stopping that process, deleting the file, then relaunching the process.

Anyone have any ideas which process is responsible for the ~/Library/Cookies/HSTS.plist file? Once we know that it should be possible to give a simpler fix to the problem.

Answer 6

You will have good results if you use the command line to curl the device to make sure it's not doing the redirection. Safari doesn't really have an engine to rewrite addresses - especially if you go into private browsing to remove any history, cookies, etc...

If you're not sure you have cleaned your safari enough, you can also test by opening system preferences and making a clean/new user account on the Mac and test the site on a totally clean version of Safari after logging out of your normal user.

Answer 7

After trying all of these solutions, what worked for me was:

• Remove all instances of the domain from Safari's history
• Quit Safari
• Delete ~/Library/Cookies/HSTS.plist
• Restart

Answer 8

Here's an idea!

You say you can't undo the redirect by setting the server to redirect https requests back to http (as you don't have admin access to do so).

But what if you trick safari into connecting to a different server that offers this reverse redirect?

You could set this up in your local machine's /etc/hosts file.

For example let's say the current cached redirect is from http://example.com to https://example.com.

Now set up or identify a url that you can request on any server in the world that redirects from https back to http. Let's say that server has the address of https://redirecting.example.com.

Then look up the IP address of redirecting.example.com. In Terminal you can do like this:

host redirecting.example.com

You get a result something like this:

redirecting.example.com has address 69.69.69.69

Now open up your /etc/hosts file and add a new line that points requests for example.com at the ip address of redirecting.example.com, like so:

### point host example.com at the ip address of redirecting.example.com
69.69.69.69 example.com

Save your changes and clear your DNS cache in terminal like so:

sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder; say DNS cache flushed

Then in Safari make a request for https://example.com the response should be a redirect back to http://example.com, at which point (fingers crossed) your Safari redirect from 6 months ago will be overwritten.

When done remove the line you added to your /etc/hosts file and flush your DNS cache again.

Answer 9

My two cents for new macOS Mojave 10.14 Beta (18A365a)

a) You cannot stop definitively nsurlstoraged, it relaunches in 2 seconds, even if sudo

b) you cannot delete "HSTS.plist ": if You type:

sudo rm -f ~/Library/Cookies/HSTS.plist

you get: Operation not permitted

c) even if you try:

ls -la ~/Library/Cookies/

you get: Operation not permitted

the same for

nano ~/Library/Cookies/HSTS.plist 

(empty file..)

So you cannot definitely access it. (maybe SIP ?)

d) strangely you can delete if from Finder:

CMD Shift G "~/Library/Cookies/"

and you can delete with mouse:

e) more strange: You can move to Desktop using mouse, edit and place it back!

(a real nonsense, GUI is more powerful than sudo..)

Answer 10

First make sure the server is not sending the Strict-Transport-Security header
You can do this with curl -I (-I just gets the headers)

curl -I http://my-http-domain.com

If the server is sending the Strict-Transport-Security header, removing it from your browser will have no effect, as next time you access the site, it will be set again.

Remove your site from Safari's Http Secure Transport Security database

1. Close Safari
2. Edit ~/Library/Cookies/HSTS.plist
   Search for the entry for the site you want to access over http and remove it, and save the file.
   • I prefer to edit rather than remove as there is no need to remove valid entries.
   • I edit plist files using Xcode, but if it's not installed you can just use a text editor.
3. Restart your computer.
   • Instead of restarting your computer, you can restart nsurlstoraged but that can be involved due to SIP so a computer restart may be simpler. See Grant's answer and Quanta's answer about restarting nsurlstoraged

Answer 11

I made a script from Grand Heaslip's answer:

#!/bin/sh

osascript -e 'quit app "Safari"'
sleep 2
killall nsurlstoraged
sleep 2
rm -f ~/Library/Cookies/HSTS.plist
launchctl start /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist

It gracefully ends safari, stops nsurlstoraged, removes the HSTS.plist, and starts nsurlstoraged again. This worked fine for me here on macOS 10.13.5

Answer 12

I'm using Mojave (10.14). I tried the methods given so far to remove HSTS.plist. In addition, I needed to add Terminal to the System Preferences > Security & Privacy > Full Disk Access list to cure the symptom "Operation not permitted" when listing the contents of ~/Library/Cookies/.

But, removing the file and restarting the daemon didn't work. So I tried opening Safari again, went to Preferences, Privacy, Manage Website Data. Then I removed all "cache cookies, local storage" for the offending domain name. That solved my problem.

I can't say now if removing HSTS was required or not.

Answer 13

Very simple as of Safari 16.2

1. Safari > Preferences > Privacy > Manage Website Data…
2. Search for the domain for which you want to reset HTTP redirections.
3. Click "Remove".
   • Note: This removes "Cache, Cookies, Local Storage, and HSTS Policy" altogether for that particular domain. There is no method to delete only the cached HTTP redirections alone.

How I determined that this really works

On a web server under my control I used Wordpress with the Redirection plugin.

1. Wordpress > Admin Menu > Tools > Redirection > Log: Is empty.

2. I intentionally trigger a HTTP 301 redirection by entering URL-1 which gets redirected to URL-1-transformed.

   👉 Server side log shows URL-1 → URL-1-transformed with the timestamp of my request.

3. I invoke the same URL-1 again 2 more times.

   👉 Server side log does not show any corresponding log entries.

   👉 This means that the web browser has used its own HTTP redirection cache.

4. Cleared HTTP redirection cache for the domain as in the 1-2-3 instruction above.

5. Again requested URL-1 3 times. Always got redirected properly.

   👉 Server side log shows URL-1 → URL-1-transformed with the timestamp of the first request of my new request series.

   👉 That timestamp for the 1st new request proves that Safari had no more client-side HTTP redirection cache for the given domain and hence asked the server which then answered with a HTTP 301 redirection response.

   👉 The 2nd and 3rd request from that series then again must have happened via the client side HTTP cache as the server side log showed no corresponding redirection log entries.

Further background infos

After step #3 or #5 searching on disk for URL-1 or URL-1-transformed in the web browsers application data folders or caching folders yields nothing.

• This cache seems to be solely in RAM.
• And only get persisted rarely or possible not at all.

Possible reasons why the browser vendors don't persist this:

• The redirection headers may have a short lifespan due their expiration-date being close.
• Possibly also for performance reasons browsers keep this solely in RAM.
• And overally due to the rather ephemer nature and potential undesired side effects the web browser developers possible do not consider it necessary/worthy to build methods to write/read this to/from persistent storage.

There are no known tools to inspect or manipulate cached HTTP redirections in:

• Safari Web Inspector
• Google Chrome DevTools — A similar thread on Google Chrome says: "It's unclear where Chrome stores the redirects."

How long to cache HTTP redirects also varies greatly across browsers.

Tested on Safari 16.2 (released 2022-12-13) on macOS 11.7.2 Big Sur (elder OS already). Appreciating confirmations of newer macOS versions.

Answer 14

Try this then, go to Step 1: Go to ~/Library folder, Step 2 :Delete Safari folder from ~/Library/Application Support, Step 3: Delete below folders from ~/Library/Caches, Step 4: then Delete ~/Library/Safari folder P.S:Keep safari closed during the above operations