38

I have a device with a web-based control panel, and accidentally set it to redirect all http pages to https, even though some don't work over https. Although I've since corrected this, Safari seems to have memorised the redirect and is refusing to forget it, instead constantly attempting to redirect me to the invalid https address.

I've already closed Safari, cleared ~/Library/Caches/com.apple.Safari/ and ~/Library/Cookies/HSTS.plist but it still seems to be remembering the redirect when I re-open it.

Where else could Safari be storing this information? I can access the correct page via Firefox or Chrome, so it may not be a system wide service, or if it is it's not one that the other browsers use.

Unfortunately because the web-panel is provided by a device I don't believe I can adjust headers or setup a redirect back to the correct URL, which seem to be options offered in other similar questions, so I really need to find out where this data is being stored so I can destroy it with fire.

7
  • See also apple.stackexchange.com/questions/153503/… Commented Nov 3, 2016 at 12:28
  • Have you tried trashing/moving aside your ~/Library/Safari folder and seeing if that fixes the problem? If it does, you can experiment with items inside the folder until you find the culprit file. Commented Dec 31, 2016 at 14:03
  • How did you set the redirect? With an extension or is there a setting in Safari for that?
    – owlswipe
    Commented Dec 31, 2016 at 15:55
  • Does the redirect still happen with a private browsing window?
    – AllInOne
    Commented Jan 4, 2017 at 20:39
  • @AllInOne interesting idea, but unfortunately it still happens under private browsing.
    – Haravikk
    Commented Jan 5, 2017 at 10:20

14 Answers 14

44

Based on quanta’s answer:

I wasn’t able to use launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist because I have System Integrity Protection enabled:

$ launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist
/System/Library/LaunchAgents/com.apple.nsurlstoraged.plist: Operation not permitted while System Integrity Protection is engaged

However I was able to work around it by doing the following:

  • killall nsurlstoraged (stops your user’s nsurlstoraged process; I actually ran sudo killall nsurlstoraged, but I suspect it isn’t necessary to stop the system’s nsurlstoraged as well, since the cache is in the user Library folder)
  • rm -f ~/Library/Cookies/HSTS.plist (deletes the HSTS cache)
  • launchctl start /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist (restarts nsurlstoraged)
6
  • I can't upvote this answer enough. It appears that at least on Sierra, simply removing the HSTS.plist file won't fix the issue because it will continue to be rebuilt. However, after killing nsurlstoraged and then removing the HSTS file—that did the trick!
    – nvahalik
    Commented Aug 28, 2017 at 16:52
  • 1
    Thanks very much, upvoted, but I did it like this. 1. Close Safari 2. Edit ~/Library/Cookies/HSTS.plist and remove the entry for the site I want on http 3. Restart computer
    – Jason S
    Commented Feb 3, 2018 at 21:42
  • Yeah, restarting is the advice that all the other answers give you, but with 20 apps open it's much more convenient and faster to restart the nsurlstoraged process. Thanks @nvahalik !
    – axello
    Commented Apr 30, 2018 at 21:38
  • 4
    Mojave update: the command rm -f ~/Library/Cookies/HSTS.plist will return Operation not permitted unless you have granted Full Disk Access to Terminal.app in System Preferences => Security & Privacy => Privacy. Otherwise, the solution worked perfectly! Thanks!
    – joehanna
    Commented Mar 4, 2019 at 6:36
  • 1
    This doesn't appear to do anything in Big Sur. The killall command returns 'No matching processes belonging to you were found' Commented Feb 4, 2021 at 13:30
10

If you enable Develop menu in Safari preferences, you can clear cache from there (CMD+ALT+E).

Can you confirm that opening the device's control panel in Safari's Private window (or different web browser) works correctly?

3
  • 1
    Unfortunately the develop menu option doesn't seem to clear the redirect, neither does closing Safari and manually deleting ~/Library/Caches/com.apple.Safari so the redirect must be getting stored somewhere else. HSTS was the feature I accidentally enabled but I've already deleted ~/Library/Cookies/HSTS.plist.
    – Haravikk
    Commented Feb 17, 2016 at 9:02
  • 1
    I can also confirm this answer doesn't fix it
    – malhal
    Commented Aug 15, 2017 at 22:02
  • 1
    This one worked for me Commented Oct 5, 2018 at 11:19
8

Based on @Haravikk's answer: https://apple.stackexchange.com/a/267783/62907

Anyone have any ideas which process is responsible for the ~/Library/Cookies/HSTS.plist file?

fs_usage can help:

❯❯❯❯ sudo fs_usage | grep HSTS
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000238   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000009   nsurlstorage
16:11:03  open              /Users/quanta/Library/Cookies/HSTS.plist                                         0.016268   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000008   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000003   nsurlstorage
16:11:03  access            /Users/quanta/Library/Cookies/HSTS.plist                                         0.000011   dbfseventsd
16:11:04  lstat64           /Users/quanta/Library/Cookies/HSTS.plist                                         0.000008   fseventsd
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000006   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000002   nsurlstorage
16:11:08  open              /Users/quanta/Library/Cookies/HSTS.plist                                         0.000144   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000002   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000003   nsurlstorage
16:11:08  access            /Users/quanta/Library/Cookies/HSTS.plist                                         0.000021   dbfseventsd
16:11:09  lstat64           /Users/quanta/Library/Cookies/HSTS.plist                                         0.000042   fseventsd

So, we can:

launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist

then:

rm -f ~/Library/Cookies/HSTS.plist

and try again.

2
  • Thanks! This worked for me. I deleted HSTS.plist many times (closing/restarting Safari before and after) and it was always being recreated with exactly the same contents as before. Unloading nsurlstoraged first, then deleting the plist and restarting nsurlstoraged gave me a clean plist.
    – lucianf
    Commented Jun 20, 2017 at 14:07
  • 2
    You could improve this by mentioning you need to quit and relaunch Safari for it to work. Also rather than delete the HSTS.plist I just deleted the problem domain key.
    – malhal
    Commented Aug 15, 2017 at 22:06
7

In Safari, Firefox and Chrome as well, all you have to do is open the developer sidebar, select the network tab, and disable caching.

Disable Caches Checkbox for 503 permanent redirects

Activate that, and the old permanent redirect should be ignored.

The greatest benefit is that you don't have to mess with files, you won't delete all HTST entries and loose the security benefits. Also, it works across browsers.


Older Safari

In older Safari, that's a crossed out tube thing, the blue one next to the trash bin logo. Safari disable Caching 503 permanent redirects

5
  • While this is very useful to know, can you please confirm whether it functions as a permanent solution? i.e- if the cache is re-enabled, will the problem resurface, or does temporarily disabling it flush it?
    – Haravikk
    Commented Jul 15, 2019 at 11:45
  • 1
    @Haravikk in my tests it would not revert to using the permanent redirect when a new page could be loaded instead. Even after closing the development window, if that answers your question Commented Jul 15, 2019 at 18:25
  • 1
    This is by far the best answer. Disabling the cache makes the browser replace the cached response for the resource, so when you enable the cache again it will not redirect.
    – PEZ
    Commented Jan 13, 2023 at 17:22
  • in 'modern' safari (i.e. in Safari 17) it's just a checkbox called 'Disable Caches' in the Network tab, and it's not some weird icon anymore -- for me this worked great!!! thanks
    – MichielB
    Commented Sep 21, 2023 at 11:42
  • Just tested on Safari 16.6 and while the cache was not used while the flag is cleared, it was reused immediately after I checked the flag again
    – Alfonso
    Commented Sep 28, 2023 at 7:34
4

So I've found a workaround to the problem, though this isn't a definitive answer to the actual question so I will not mark it as such until I can find more information.

It turns out that the file ~/Library/Cookies/HSTS.plist was indeed the source of the problem as I suspected, however deleting it from the affected user account doesn't work, even with Safari closed, as it is recreated after an unknown amount of time, complete with the offending entry that was forcing the invalid redirect.

So my solution was the following:

  1. Make sure you have at least one other user account on your Mac (if not, create one).
  2. Logout of the affected user account.
  3. Login to a different user account (a guest account may not be sufficient, depending upon restrictions).
  4. Find out the short-name of your affected user account; if you don't know then the best way to check is to look under System Preferences -> Users. Usually if will be the full name, lower-cased and with no spaces, so if your full name is "John Smith" then the short-name may be "johnsmith".
  5. Open a window in Terminal, type su shortname replacing "shortname" with the short-name of the affected user-account. Hit enter and, when prompted, enter the password for the affected account.
  6. Now type the next command rm ~/Library/Cookies/HSTS.plist and hit enter, this will delete the HSTS storage file.
  7. Finally type exit, hit enter and close Terminal.

At this point you can now log back into the affected user account and the offending HSTS redirect should be gone for good.

Now, while this provides a usable workaround, I'd really like to know why deleting the HSTS.plist file from my affected account didn't work; the fact it is recreated means some background process is responsible for it, which means it should be possible to delete the file from the affected user account by simply stopping that process, deleting the file, then relaunching the process.

Anyone have any ideas which process is responsible for the ~/Library/Cookies/HSTS.plist file? Once we know that it should be possible to give a simpler fix to the problem.

3

You will have good results if you use the command line to curl the device to make sure it's not doing the redirection. Safari doesn't really have an engine to rewrite addresses - especially if you go into private browsing to remove any history, cookies, etc...

If you're not sure you have cleaned your safari enough, you can also test by opening system preferences and making a clean/new user account on the Mac and test the site on a totally clean version of Safari after logging out of your normal user.

2
  • There's definitely no redirect (the feature I'm trying to connect to doesn't support HTTPS at all, which is why enabling HSTS for the entire device was a terrible, terrible mistake); I can connect just fine from other user accounts and browsers, so there's something stored somewhere on my main account that is caching this :(
    – Haravikk
    Commented Feb 17, 2016 at 9:03
  • “Safari doesn't really have an engine to rewrite addresses” — I’ve currently got the same problem occurring in Safari with a website hosted on my laptop, and curl (along with Firefox, Chrome, and a Private Browsing window right there in Safari) on the same user account loads the site just fine. So it must be something to do with Safari itself. Commented Nov 3, 2016 at 11:42
3

After trying all of these solutions, what worked for me was:

  • Remove all instances of the domain from Safari's history
  • Quit Safari
  • Delete ~/Library/Cookies/HSTS.plist
  • Restart
1
  • This is the correct answer to clear 301 redirect cache. None of the above work unless you also remove instances of the domain from Safari's history.
    – Dustin
    Commented Feb 5, 2020 at 17:32
2

Here's an idea!

You say you can't undo the redirect by setting the server to redirect https requests back to http (as you don't have admin access to do so).

But what if you trick safari into connecting to a different server that offers this reverse redirect?

You could set this up in your local machine's /etc/hosts file.

For example let's say the current cached redirect is from http://example.com to https://example.com.

Now set up or identify a url that you can request on any server in the world that redirects from https back to http. Let's say that server has the address of https://redirecting.example.com.

Then look up the IP address of redirecting.example.com. In Terminal you can do like this:

host redirecting.example.com

You get a result something like this:

redirecting.example.com has address 69.69.69.69

Now open up your /etc/hosts file and add a new line that points requests for example.com at the ip address of redirecting.example.com, like so:

### point host example.com at the ip address of redirecting.example.com
69.69.69.69 example.com

Save your changes and clear your DNS cache in terminal like so:

sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder; say DNS cache flushed

Then in Safari make a request for https://example.com the response should be a redirect back to http://example.com, at which point (fingers crossed) your Safari redirect from 6 months ago will be overwritten.

When done remove the line you added to your /etc/hosts file and flush your DNS cache again.

3
  • While it's a nice idea it doesn't solve the real problem; I'm not looking for workarounds, but rather I want to know where this redirect is being cached such that Safari continues to use it even though it is no longer valid (the server does not have HSTS enabled, I simply enabled it briefly by mistake). It must be stored somewhere, but I can't figure out where.
    – Haravikk
    Commented Jan 6, 2017 at 18:29
  • This is not what I would call a workaround, as I expect it will solve the actual problem. It only works around the fact that you don't have control of the device. But I do hear you -- it would be nice to be able to clear the cached setting directly. Does Safari Technology Preview also exhibit the bad behavior?
    – AllInOne
    Commented Jan 6, 2017 at 18:34
  • Unfortunately so; I don't think it's a problem with Safari itself as such, but rather some macOS service that it depends upon, as it does indeed appear that ~/Library/Cookies/HSTS.plist was the culprit, but deleting it from the affected account doesn't work (as it is recreated some time later, complete with bad redirect). Not sure what process is doing it though.
    – Haravikk
    Commented Jan 7, 2017 at 13:45
2

My two cents for new macOS Mojave 10.14 Beta (18A365a)

a) You cannot stop definitively nsurlstoraged, it relaunches in 2 seconds, even if sudo

b) you cannot delete "HSTS.plist ": if You type:

sudo rm -f ~/Library/Cookies/HSTS.plist

you get: Operation not permitted

c) even if you try:

ls -la ~/Library/Cookies/

you get: Operation not permitted

the same for

nano ~/Library/Cookies/HSTS.plist 

(empty file..)

So you cannot definitely access it. (maybe SIP ?)

d) strangely you can delete if from Finder:

CMD Shift G "~/Library/Cookies/"

enter image description here

and you can delete with mouse:

enter image description here

e) more strange: You can move to Desktop using mouse, edit and place it back!

(a real nonsense, GUI is more powerful than sudo..)

1
1

First make sure the server is not sending the Strict-Transport-Security header
You can do this with curl -I (-I just gets the headers)

curl -I http://my-http-domain.com

If the server is sending the Strict-Transport-Security header, removing it from your browser will have no effect, as next time you access the site, it will be set again.

Remove your site from Safari's Http Secure Transport Security database

  1. Close Safari
  2. Edit ~/Library/Cookies/HSTS.plist
    Search for the entry for the site you want to access over http and remove it, and save the file.
    • I prefer to edit rather than remove as there is no need to remove valid entries.
    • I edit plist files using Xcode, but if it's not installed you can just use a text editor.
  3. Restart your computer.
    • Instead of restarting your computer, you can restart nsurlstoraged but that can be involved due to SIP so a computer restart may be simpler. See Grant's answer and Quanta's answer about restarting nsurlstoraged
1

I made a script from Grand Heaslip's answer:

#!/bin/sh

osascript -e 'quit app "Safari"'
sleep 2
killall nsurlstoraged
sleep 2
rm -f ~/Library/Cookies/HSTS.plist
launchctl start /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist

It gracefully ends safari, stops nsurlstoraged, removes the HSTS.plist, and starts nsurlstoraged again. This worked fine for me here on macOS 10.13.5

1

I'm using Mojave (10.14). I tried the methods given so far to remove HSTS.plist. In addition, I needed to add Terminal to the System Preferences > Security & Privacy > Full Disk Access list to cure the symptom "Operation not permitted" when listing the contents of ~/Library/Cookies/.

But, removing the file and restarting the daemon didn't work. So I tried opening Safari again, went to Preferences, Privacy, Manage Website Data. Then I removed all "cache cookies, local storage" for the offending domain name. That solved my problem.

I can't say now if removing HSTS was required or not.

1
  • I've tried the same and rebooting twice, but only using Safari's UI worked for me as well. Thank you! Commented Aug 23, 2019 at 3:49
1

Very simple as of Safari 16.2

  1. Safari > Preferences > Privacy > Manage Website Data…
  2. Search for the domain for which you want to reset HTTP redirections.
  3. Click "Remove".
    • Note: This removes "Cache, Cookies, Local Storage, and HSTS Policy" altogether for that particular domain. There is no method to delete only the cached HTTP redirections alone.

How I determined that this really works

On a web server under my control I used Wordpress with the Redirection plugin.

  1. Wordpress > Admin Menu > Tools > Redirection > Log: Is empty.

  2. I intentionally trigger a HTTP 301 redirection by entering URL-1 which gets redirected to URL-1-transformed.

    👉 Server side log shows URL-1 → URL-1-transformed with the timestamp of my request.

  3. I invoke the same URL-1 again 2 more times.

    👉 Server side log does not show any corresponding log entries.

    👉 This means that the web browser has used its own HTTP redirection cache.

  4. Cleared HTTP redirection cache for the domain as in the 1-2-3 instruction above.

  5. Again requested URL-1 3 times. Always got redirected properly.

    👉 Server side log shows URL-1 → URL-1-transformed with the timestamp of the first request of my new request series.

    👉 That timestamp for the 1st new request proves that Safari had no more client-side HTTP redirection cache for the given domain and hence asked the server which then answered with a HTTP 301 redirection response.

    👉 The 2nd and 3rd request from that series then again must have happened via the client side HTTP cache as the server side log showed no corresponding redirection log entries.

Further background infos

After step #3 or #5 searching on disk for URL-1 or URL-1-transformed in the web browsers application data folders or caching folders yields nothing.

  • This cache seems to be solely in RAM.
  • And only get persisted rarely or possible not at all.

Possible reasons why the browser vendors don't persist this:

  • The redirection headers may have a short lifespan due their expiration-date being close.
  • Possibly also for performance reasons browsers keep this solely in RAM.
  • And overally due to the rather ephemer nature and potential undesired side effects the web browser developers possible do not consider it necessary/worthy to build methods to write/read this to/from persistent storage.

There are no known tools to inspect or manipulate cached HTTP redirections in:

  • Safari Web Inspector
  • Google Chrome DevTools — A similar thread on Google Chrome says: "It's unclear where Chrome stores the redirects."

How long to cache HTTP redirects also varies greatly across browsers.

Tested on Safari 16.2 (released 2022-12-13) on macOS 11.7.2 Big Sur (elder OS already). Appreciating confirmations of newer macOS versions.

-1

Try this then, go to Step 1: Go to ~/Library folder, Step 2 :Delete Safari folder from ~/Library/Application Support, Step 3: Delete below folders from ~/Library/Caches, Step 4: then Delete ~/Library/Safari folder P.S:Keep safari closed during the above operations

1
  • 1
    Answers on Ask Different need to be more than just a link. It's okay to include a link, but please summarize or excerpt it in the answer. The idea is to make the answer stand alone.
    – nohillside
    Commented Jan 6, 2017 at 10:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .