How to store proxy credentials on macOS so they are used by system services?

Question

I'm using macOS Sierra 10.12.6 behind a corporate NTLM proxy. My browser and other applications are using the system proxy settings, in which I have saved my username and password for authenticating with the proxy. This is working fine.

There is a persistent problem with system services that try to access information on the internet and don't see to have access to the proxy credentials in my user account. I see the following popup every couple of minutes, and whatever I do (updating my credentials in System Preferences, or pressing "Not Now"), the popup keeps coming up again and again:

The text in the popup reads:

Proxy Authentication Required

Enter the password for the HTTP proxy http://xxx.xxx.xxx.xxx:yyyy in System Preferences.

What can I do to stop this popup from appearing?

Things I have tried so far:

• Updated my credentials in System Preferences (Network > Advanced > Proxy)
• Copied the credentials entries from my login keychain to the System keychain, since I read a recommendation for that in a blog post or forum question.

None of these has worked, I get this popup every couple of minutes, and there does not seem to be a pattern to when it shows up.

Update 1:

As soon as I enter my credentials by clicking the System Preferences button in the above dialog (which I can force by e.g. opening Safari and starting to type a URL in the location box), two records are created in the login keychain, both with identical content:

@ xxx.xxx.xxx.xxx (username) Internet Password Today, 09:10 -- login

Both records look identical, with the same name and attributes. Both show that the application that requested this is AuthBrokerAgent:

Update 2:

I've also tried this suggestion: https://discussions.apple.com/message/23848961#message23848961, copying the authentication entries from the login keychain to the system keychain and then rebooting, but it did not fix it. In fact, the dreaded "Proxy Authentication Required" box showed up again while typing this...

Update 3:

I've used Wireshark to take a look at the traffic between my machine and our proxy:

• The proxy returns with a 407 Proxy Authentication Required and Proxy-Authenticate: NTLM, which is in line with my expectation, since our proxy uses NTLM.
• Some examples I've seen in the traffic (e.g. iCloud) then send back an NTLMSSP_NEGOTIATE response.
• The proxy comes back with an NTLMSSP_CHALLENGE request
• The service responds with NTLMSSP_AUTH and my username, which it must have gotten from somewhere.
• The proxy finally responds with a 200 Connection established

To me, this shows that in general the proxy authentication works fine, if the system can get the username and proxy from somewhere. The question remains how/where to store the username/password so that all system services can find it. Some system services (I assume) don't have any means to find the proxy credentials where I'm currently storing them.

Answer 1

This is most likely expected behavior if your system/network administrator has configured the proxy force authentication that requires more than just a basic authentication scheme.

From Microsoft's page Handling Authentication under About HTTP Authentication section:

There are two general types of authentication schemes:

• Basic authentication scheme, where the user name and password are sent in cleartext to the server.
• Challenge-response schemes, which allow for a challenge-response format.

Challenge-response schemes enable more secure authentication. If a request requires authentication using a challenge-response scheme, the appropriate status code and Authenticate headers are returned to the client. The client must then to resend the request with a negotiate. The server would return an appropriate status code with a challenge, and the client would then require to resend the request with the proper response to get the requested service.

If the proxy you are using utilizes the basic authentication scheme, what's saved in your keychain will suffice to authenticate you. If a challenge response scheme is being used, you will most likely have to provide more info - in this case - re-enter your password - to authenticate; and this is what you are seeing.

NTLM Authentication Process

This is much more than just storing credentials. The client must generate a response based upon a generated request from the server. Following is a very abridged description of the authentication process from the client/server perspective per Microsoft's documentation

• The client sends the user name to the server (in plaintext).

• The server generates a 16-byte random number, called a challenge or nonce, and sends it to the client.

• The client encrypts this challenge with the hash of the user's password and returns the result to the server. This is called the response.

• The server sends the following three items to the domain controller:

  • User name
  • Challenge sent to the client
  • Response received from the client

• The DC validates the encrypted challenge and response. If authenticated, access is granted.

The third step above, requires the client to hash a random number it got from the server. This inherently means there's nothing to be stored on your macOS client.

At a very minimum, you need to be joined the Active Directory domain. This means you need Kerberos support enabled and configured properly for your specific organization.

There's a key phrase in the "Handling Authentication" document I linked above:

If authentication is required, the INTERNET_FLAG_KEEP_CONNECTION flag should be used in the call to HttpOpenRequest. The INTERNET_FLAG_KEEP_CONNECTION flag is required for NTLM and other types of authentication in order to maintain the connection while completing the authentication process. If the connection is not maintained, the authentication process must be restarted with the proxy or server.

^{(Emphasis mine)}

Based on your the symptoms being presented it appears that your organization requires authentication to the proxy; your username/password are valid, but it keeps (re)asking for authentication. It's probably because you are losing the connection state and having to do this all over again. Which further emphasizes the point....

To solve this issue, you will need to contact your network admin to assist you with the authentication issues.

Answer 2

by authention of key chain app modify setting

Answer 3

Run the following command from Console.app:

networksetup -setwebproxy "Your Interface Name" "web proxy hostname or IP" 
8080 on username password

You will be asked about keychain access. Agree to add the record into keychain and you will have an access without the password all the time when your keychain is open.