How to use pip after the OS X El Capitan upgrade?

Question

After the El Capitan update, I am unable to run pip install. The error I get is that the "operation is not permitted" when pip tries to create new folders while installing.

creating /System/Library/Frameworks/Python.framework/Versions/2.7/share
    error: could not create '/System/Library/Frameworks/Python.framework/Versions/2.7/share': Operation not permitted

In fact, in general I cannot create folders etc in these folders. Have tried using sudo that doesn't help. Also I have done

sudo chflags nouchg /System/Library/Frameworks/Python.framework/Versions/2.7/

The above didn't help either. What can I do to be able to install python libraries smoothly again?

Answer 1

A quick solution is to use homebrew to install python into /usr/local/bin so that your pip can run against a user-modifiable python framework.

brew install python
pip --version

Disabling System Integrity Protection is also an option, but I don't recommend that for anything but professionally managed and fire walled servers where you have the manpower to manage intrusion detection or if you are a developer/sysadmin and need to test things with and without SIP.

ls -lO /System/Library/Frameworks/Python.framework/Versions/2.7/
csrutil status

You will see that the restricted flag is set which cannot be removed even as root while SIP is engaged.

Using homebrew makes it possible to manage pip and python separately than the system provided version. As a bonus, the homebrew framework is designed to ease maintainance and patch/chores via automation.

Answer 2

Another viable option without a need to disable SIP or install other Python versions, is to install the modules only for the current user using

pip install --user <modulename>

If it is just your personal machine, this would be the simplest and safest solution.

Answer 3

This problem often arises when pip tries to install a manpage for IPython on El Capitan. The quick fix is to use a pip command like this:

sudo -H pip install --install-option '--install-data=/usr/local' <package>

However, System Integrity Protection (SIP) on El Capitan blocks several bad practices with pip that used to slide by, so you will probably need to make some more changes to get pip running smoothly on El Capitan.

SIP on El Capitan exposes three problems with using pip with the Apple-supplied version of Python on OS X:

1. distutils does not set two important variables correctly on Macs, so pip tries to write headers and other shared files (e.g., manpages) under /System/Library/Frameworks/Python.framework/Versions/2.7/. This is a bad idea, but in earlier versions of OS X, it succeeded if pip was run with sudo. However it fails on El Capitan due to SIP. This is the error you ran into. It gives messages like OSError: [Errno: 1] Operation not permitted: '/System/Library/Frameworks/Python.framework/Versions/2.7/share'

2. Apple installs outdated versions of some packages in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/ (e.g., six). On previous versions of OS X, when you installed a package that needed a newer version of one of these, sudo pip would silently remove the old version from the /System/ folder and install a newer version in /Library/Python/2.7/site-packages. This was also a bad idea, and is no longer possible with SIP. But now pip will crash with an error message while trying to remove the old package. That message is also OSError: [Errno: 1], but it comes after a message like Uninstalling six-1.4.1:. See, e.g., https://github.com/pypa/pip/issues/3165 .

3. The Apple version of Python adds several directories under /System/Library/Frameworks/Python.framework/Versions/2.7/ to the python search path above the standard user-accessible package installation locations. So if you install a newer version of a package elsewhere (e.g., sudo -H pip install --ignore-installed six), you will get a message that the installation succeeded, but then when you run python you will get the older version from /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/. This also makes it impossible to use new packages that have the same name as modules from the standard library.

You can work around these issues, but the method depends on your answers to three questions.

1. Do you want to continue using the Mac OS X version of Python or install your own? Installing your own is the safest option, and can be done via the official Python installer, Homebrew or Anaconda. This is also what Apple recommends, as pointed out by @Sacrilicious. If you install your own version of Python, you should probably uninstall anything currently installed in /Library/Python/2.7/site-packages and any scripts that were installed in /usr/local/bin for those packages (including pip). Otherwise you will have the annoying experience of some scripts accessing the system-installed version of Python and some accessing your own installation.

If you want to stick with the system-installed Python, then you have to make two more decisions:

2. Do you want to install packages for all users, or just for yourself? Installing for all users ensures that every program that uses Python (including possibly administrative scripts) will have access to all the packages you install. However, there's a distant chance that it will interfere with El Capitan's own use of Python. (I would hope that Apple uses python -S to ensure they always get the packages they expect, but I have no way to test this.) Installing just for your own user account eliminates the possibility of interfering with the system Python installation. Note: if you are going to switch from system-wide installation to user-only, you should probably take this chance to uninstall anything currently installed in /Library/Python/2.7/site-packages and related scripts in /usr/local/bin.

3. Do you want to hide the extra packages that are installed with the OS X version of Python (under /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/), or keep them in the search path? I recommend hiding them, so that the newest versions of these packages will automatically be installed in user-accessible locations when needed. If you don't hide this directory, then you will occasionally get messages that pip was unable to remove an existing package in order to upgrade it to a later version (needed by a different package you are installing). In that case, you will need to run pip install --ignore-installed <package>, which will install the newer version and hide the system-installed version. However, if you hide the whole /System/.../Extras/... directory, you will lose access to some Apple packages that are not available via pip, i.e., CoreGraphics and bonjour. (If you need these, you may be able to get access by symlinking them into your site-packages directory.)

Now, here are the workarounds. These would be good practice on all versions of OS X, to avoid accidentally replacing or removing Python packages used by the operating system; however they are essential if you want to use user-installed packages with the Apple-supplied version of Python on OS X El Capitan (10.11).

Install pip

You probably did this already, but if not, you can use the following command to install pip for all users:

sudo -H easy_install pip
# pip script will be installed in /usr/local/bin

Or use this command to install pip for your own user account only:

easy_install --user pip
# pip script will be installed in ~/Library/Python/2.7/bin

Manage Shared File Locations

If you are installing packages for all users, create a file called .pydistutils.cfg with these lines (from https://github.com/pypa/pip/issues/426):

[install]
install-data=/usr/local
install-headers=/usr/local

If you usually use sudo -H pip ..., then you should put this file in /var/root (home directory for the root user). If you usually use sudo pip ..., then you should put this file in your own home directory (~).

These settings will prevent pip from trying to write shared items like headers and manpages under /Library/System. (The command at the top of this answer is a quicker version of the same thing.) These settings are needed because darwin-specific code in /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/install.py fails to set these variables to root-writeable locations (although it sets other variables correctly). There is more information on this at https://github.com/pypa/pip/issues/3177 .

If you install packages for your own user account only, shared items will automatically be installed under ~/Library/Python/2.7/. But you should add the following lines to your ~/.profile so the shared items will be found when you need them:

export PATH=~/Library/Python/2.7/bin:$PATH
export MANPATH=~/Library/Python/2.7/share/man:$MANPATH

Note: you will need to start a new shell or run these on the command line for the changes to take effect. You may also want to run hash -r if you've recently removed old scripts from the path.

Manage Python Path

You will need to ensure that the packages you install are higher in Python's search order than system-installed packages. The easiest way to do this is with .pth files. This follows @Sacrilicious's suggestion elsewhere on this page, but ensures that your user site-packages directory is searched before your system-wide site-packages directory, and both are searched before the standard library and Apple's Extras directory (both under /System/...). It also omits /System/.../Extras from the search path if desired.

Create a file called fix_mac_path.pth, with the text below. If you are installing packages for all users, fix_mac_path.pth should be placed in /Library/Python/2.7/site-packages. If you are installing only for your own user, fix_mac_path.pth should be in ~/Library/Python/2.7/lib/python/site-packages. (This file can have any name you want, but it has to be placed in one or both of these locations, and it has to end with .pth; also, all the text in this file has to be on one line.)

If you want to hide the Apple-installed packages in /System/.../Extras:

First run one of the following commands to get a working copy of pip/setuptools independent from the Apple-supplied version:

pip install --ignore-installed --user setuptools   # your account only
# or
sudo -H pip install --ignore-installed setuptools  # all users

Then put the following code in fix_mac_path.pth at the location specified above:

import sys; std_paths=[p for p in sys.path if p.startswith('/System/') and not '/Extras/' in p]; sys.path=[p for p in sys.path if not p.startswith('/System/')]+std_paths

If you want to keep using the Apple-installed packages, you don't need to install another copy of setuptools. Just put the following code in fix_mac_path.pth at the location specified above:

import sys; std_paths=[p for p in sys.path if p.startswith('/System/')]; sys.path=[p for p in sys.path if not p.startswith('/System/')]+std_paths

After this, you can use python -m site to make sure the path search order makes sense.

Install Packages

After this, you should be able to install new packages using one of the following commands.

For all users:

sudo -H pip install <package>

For your own user:

pip install --user <package>

Answer 4

First, you're not disabling SIP as a way to address the issue. Sorry that's the reason that this folder creation is failing, but we have to work around it. Second, you lose all the niceties Apple thought they were giving you, like a bridge via pyObjC, when you install your own Python.
(I'll admit, they say you should install your own to /usr/local if you're a developer, I just object to doing it the easy way with brew.)

Logically, you'd think to install it somewhere SIP isn't blocking you, and this is almost certainly some dependency pip is resolving for you. Telling pip where you want it to perform installs isn't the remedy as much as having it ignore the pre-existing stuff in /System, which is where it would check for dependencies for many common installs that need, e.g.six, and SIP causes pip to break itself when it goes to try and upgrade it. It's actually pip's default behavior to install to /Library/Python/2.7/site-packages, you just need to say --ignore-installed to force it to install any updated versions of dependencies there.
...Unfortunately, you may then go to import this module in an interactive session or script and get nice fat failures - it's still looking at /System's un-upgraded package!

Since early setuptools days, Python has a way to make that lookup explicit. It's a bit odd-looking, (and should elicit pause to those of us who are security-conscious and not thrilled about the bluntness of shoving it to the top of the list without checking, but) you can place a file that ends with .pth (e.g. 'elcap.pth') in /Library/Python/2.7/site-packages to push that directory to the front of the lookup order with the following contents:
import sys; sys.path = ['/Library/Python/2.7/site-packages'] + sys.path

Start a new session, and a quick trip to python -m site will confirm that you've inserted that path into the first slot, and importing modules should work.

Oh, and after all this, try just installing with pip's --user option, or using a virtualenv - that's best practice for most folks anyway.

Answer 5

Believe me, you don't really want the library to write anything at that path.

It was previously not recommended, but possible to write into /System/Library/Frameworks/Python.framework/Versions/2.7/ , but now it's not supported due to Apple SIP and so it's the library owner problem. The package distribution should be upgraded to work correctly with this update. Most packages were updated and install their content to /Library/Python/2.7/site-packages, but some packages were not updated.

For my case, it was greenlet library trying to write its .h file into System Frameworks folder:

How to fix it: sudo -H pip install greenlet --install-option "--install-headers=/Library/Python/2.7/lib/python/includes/" then sudo -H pip install gevent

For numpy, the fix is sudo -H pip install --ignore-installed -U numpy.

For other libraries, fixes vary from https://github.com/pypa/pip/issues/3177 to pip install --ignore-installed six and pip install --user (last one installs everything into /User//Library/Python/2.7/ path). See also current top answer to this post: https://apple.stackexchange.com/a/210021/169157

If you type python -m site it should include sys.path = [ ... '/Library/Python/2.7/site-packages', ... ] before system paths -- that's why (and how) it works.

Answer 6

I think its because of the SIP or System Integrity Protection, a kind of real-time file protection that feels like a windows anti-virus :) and stops any changes you want to make in the OS X system folders. Apple simply decided to disable system modifications – to be sure you are fully protected from the evil of internet and other computer related evil forces SIP is protecting us from.

If you want to disable SIP you need to boot into Recovery HD by holding Command + R keys simultaneously as you boot your Mac.

Open the terminal from the Utilities menu, and type in csrutil disable, then press return. Go to the Apple Menu to reboot.

SIP would be disabled from now on. I disabled it long time ago and haven't noticed anything wrong, some processes seems to complain but they always do on OS X so i'm not sure if it's caused by the lack of System Integrity 'protection'

My tip is to entirely get rid of SIP and/or use homebrew for all of your opensource/development software. Homebrew uses the /usr/local directory for it's installment and is not colliding with other system components, and homebrew builds are more up-to-date then the os x apple builds.

Answer 7

Used pip3 install <package> instead and solved the permission problem in pip.

Answer 8

I did it as follows:

brew install python

then after python is installed:

sudo easy_install pip

Answer 9

A very cool solution for this issue is using virtualenv (virtualenvwrapper), after you create a new environment for your project, you can use pip without issues, so I used virtualenvwrapper and these two lines fix the problem:

mkproject <project_name>
pip install <package_name>

Answer 10

I have python3 installed on my mac while the original python2.7 comes with the OSX. So whenever I want to invoke the python3, i just type $python3 .... Maybe you should try $python3 get-pip.py, this works for me when I have almost the same problem with you.

Answer 11

Summary

I hit this problem on Mac OS X 10.11.6 (which has SIP) because I installed pip using the System easy_install, and the local easy-install.pth referred to system libraries.
When installing subsequent packages with pip some of the dependencies were resolved to the older System Python libraries.
The solution was to remove my locally installed libraries and reinstall a local python version (which includes pip) from https://www.python.org/downloads/ so that the System and locally installed python instances are kept separate.
I had tried using the fix_mac_path.pth fix in the answer by @mfripp however I found removing and reinstalling cleaner.

Cause

(Don't follow these steps)

I hit this problem whilst trying to install Ansible. I followed the Ansible docs for installation on OS X via pip
First I installed pip with sudo easy_install pip
This used the System easy_install at /usr/bin/easy_install and installed pip at /Library/Python/2.7/site-packages/pip
I got warnings as follows when installing pip but I ignored them and ploughed blindly on.

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
warning: no previously-included files found matching '.coveragerc'
...
Adding pip 9.0.1 to easy-install.pth file
Installing pip script to /usr/local/bin
...
Installed /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg
...
Finished processing dependencies for pip

Later looking at /Library/Python/2.7/site-packages/easy-install.pth, it looked like this

import sys; sys.__plen = len(sys.path)
./pip-9.0.1-py2.7.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

Then I installed ansible via pip

sudo pip install ansible

I got messages to say that requirements were already fulfilled by system libraries

Requirement already satisfied: setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from ansible)  
Requirement already satisfied: six>=1.4.1 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from cryptography>=1.1->paramiko->ansible)

Then when running ansible I got this problem

VersionConflict: (setuptools 1.1.6 (/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python), Requirement.parse('setuptools>=11.3'))

and a pip check revealed paramiko required a newer setuptools

pip check paramiko
matplotlib 1.3.1 requires tornado, which is not installed.
matplotlib 1.3.1 requires nose, which is not installed.
cryptography 1.7.2 has requirement setuptools>=11.3, but you have setuptools 1.1.6.

Note that ansible only required setuptools (no version) and so pip reported the dependency satisfied by the System setuptools.

Solution

I resolved it by uninstalling any local python libraries using the process hinted at https://docs.python.org/2.7/using/mac.html#getting-and-installing-macpython
For me this involved

sudo rm -rf /Library/Python

Then I removed the symlinks and executables in /usr/local/bin such as

sudo rm /usr/local/bin/ansible  # executable
sudo rm /usr/local/bin/python*  # symlinks to /Library/Python/2.7
sudo rm /usr/local/bin/easy_install*

and so on. I also removed any applications

sudo rm -rf /Applications/Python\ 2.7/

I then downloaded the 2.7.13 installer package for Mac OS X from https://www.python.org/downloads/ and installed it.

This installed a local python and pip at /Library/Frameworks/Python.framework/Versions/2.7 and symlinks in /usr/local/bin which is separate from the system libraries at /System/Library/Frameworks/Python.framework and /usr/bin so that I get

which pip
/Library/Frameworks/Python.framework/Versions/2.7/bin/pip
which ansible
/Library/Frameworks/Python.framework/Versions/2.7/bin/ansible

and in pip list

setuptools (28.8.0)
six (1.10.0)

ansible now works for me

Answer 12

For me, the PATH had gone missing for pip, this was confirmed by running python -m pip

Two potential solutions here, add pip back to PATH. In my case pip was out of date anyways, so upgrading fixed it:

python -m pip install --upgrade pip

✗ pip --version pip 9.0.1 from /Library/Python/2.7/site-packages (python 2.7)