115

After the El Capitan update, I am unable to run pip install. The error I get is that the "operation is not permitted" when pip tries to create new folders while installing.

creating /System/Library/Frameworks/Python.framework/Versions/2.7/share
    error: could not create '/System/Library/Frameworks/Python.framework/Versions/2.7/share': Operation not permitted

In fact, in general I cannot create folders etc in these folders. Have tried using sudo that doesn't help. Also I have done

sudo chflags nouchg /System/Library/Frameworks/Python.framework/Versions/2.7/

The above didn't help either. What can I do to be able to install python libraries smoothly again?

  • 1
    Where is pip on your system? – user151019 Oct 6 '15 at 22:07
  • $ where pip /usr/local/bin/pip – web_ninja Oct 6 '15 at 22:28
  • $ pip --version pip 7.1.2 from /Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg (python 2.7) – web_ninja Oct 6 '15 at 22:29
  • How did you install pip - it should write to /Library/Python/2.7/site-packages not /System – user151019 Oct 6 '15 at 23:06
  • I don't remember now. I figured after posting the comment that was the problem and used brew to install python and that corrected the location of pip. – web_ninja Oct 6 '15 at 23:12

12 Answers 12

84

A quick solution is to use homebrew to install python into /usr/local/bin so that your pip can run against a user-modifiable python framework.

brew install python
pip --version

Disabling System Integrity Protection is also an option, but I don't recommend that for anything but professionally managed and fire walled servers where you have the manpower to manage intrusion detection or if you are a developer/sysadmin and need to test things with and without SIP.

ls -lO /System/Library/Frameworks/Python.framework/Versions/2.7/
csrutil status

You will see that the restricted flag is set which cannot be removed even as root while SIP is engaged.

Using homebrew makes it possible to manage pip and python separately than the system provided version. As a bonus, the homebrew framework is designed to ease maintainance and patch/chores via automation.

  • 1
    The brew installed python is not stable and it crashes randomly – jayatubi Oct 15 '15 at 4:38
  • @jayatubi What package or script is unstable for you? Are you installing python 2.7.10 from brew or the python3 that just got a major bump and is known to not be so backward compatible. You can select from dozens of versions if you prefer one that's more or less stable for your needs. – bmike Oct 15 '15 at 14:47
  • 1
    Agree that brew install python is the simplest option, though cleaning up some other setup may be required if you've been using the system Python for some time (e.g. root-owned Python files under /usr/local/bin and ~/Library/Caches/pip) – RichVel Aug 20 '16 at 7:50
  • Downloading a compiled python .pkg from e.g. python.org/downloads/release/python-2712 works fine for me, on 10.8. Any comments on that vs. homebrew ? Thanks – denis Sep 3 '16 at 13:44
  • @denis I like brew since it is easy to pin a version you like - manage multiple different versions or keep things updated regularly. If you have something that works for you, though - it's hard to argue you should change anything. – bmike Dec 3 '16 at 0:40
77

Another viable option without a need to disable SIP or install other Python versions, is to install the modules only for the current user using

pip install --user <modulename>

If it is just your personal machine, this would be the simplest and safest solution.

  • 1
    Maybe El Capitan provided /usr/bin/pip, but macOS Sierra does not. – sigjuice Nov 14 '16 at 19:26
  • 1
    Works on sierra – harryparkdotio Feb 16 '17 at 5:50
57
+200

This problem often arises when pip tries to install a manpage for IPython on El Capitan. The quick fix is to use a pip command like this:

sudo -H pip install --install-option '--install-data=/usr/local' <package>

However, System Integrity Protection (SIP) on El Capitan blocks several bad practices with pip that used to slide by, so you will probably need to make some more changes to get pip running smoothly on El Capitan.

SIP on El Capitan exposes three problems with using pip with the Apple-supplied version of Python on OS X:

  1. distutils does not set two important variables correctly on Macs, so pip tries to write headers and other shared files (e.g., manpages) under /System/Library/Frameworks/Python.framework/Versions/2.7/. This is a bad idea, but in earlier versions of OS X, it succeeded if pip was run with sudo. However it fails on El Capitan due to SIP. This is the error you ran into. It gives messages like OSError: [Errno: 1] Operation not permitted: '/System/Library/Frameworks/Python.framework/Versions/2.7/share'

  2. Apple installs outdated versions of some packages in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/ (e.g., six). On previous versions of OS X, when you installed a package that needed a newer version of one of these, sudo pip would silently remove the old version from the /System/ folder and install a newer version in /Library/Python/2.7/site-packages. This was also a bad idea, and is no longer possible with SIP. But now pip will crash with an error message while trying to remove the old package. That message is also OSError: [Errno: 1], but it comes after a message like Uninstalling six-1.4.1:. See, e.g., https://github.com/pypa/pip/issues/3165 .

  3. The Apple version of Python adds several directories under /System/Library/Frameworks/Python.framework/Versions/2.7/ to the python search path above the standard user-accessible package installation locations. So if you install a newer version of a package elsewhere (e.g., sudo -H pip install --ignore-installed six), you will get a message that the installation succeeded, but then when you run python you will get the older version from /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/. This also makes it impossible to use new packages that have the same name as modules from the standard library.

You can work around these issues, but the method depends on your answers to three questions.

  1. Do you want to continue using the Mac OS X version of Python or install your own? Installing your own is the safest option, and can be done via the official Python installer, Homebrew or Anaconda. This is also what Apple recommends, as pointed out by @Sacrilicious. If you install your own version of Python, you should probably uninstall anything currently installed in /Library/Python/2.7/site-packages and any scripts that were installed in /usr/local/bin for those packages (including pip). Otherwise you will have the annoying experience of some scripts accessing the system-installed version of Python and some accessing your own installation.

If you want to stick with the system-installed Python, then you have to make two more decisions:

  1. Do you want to install packages for all users, or just for yourself? Installing for all users ensures that every program that uses Python (including possibly administrative scripts) will have access to all the packages you install. However, there's a distant chance that it will interfere with El Capitan's own use of Python. (I would hope that Apple uses python -S to ensure they always get the packages they expect, but I have no way to test this.) Installing just for your own user account eliminates the possibility of interfering with the system Python installation. Note: if you are going to switch from system-wide installation to user-only, you should probably take this chance to uninstall anything currently installed in /Library/Python/2.7/site-packages and related scripts in /usr/local/bin.

  2. Do you want to hide the extra packages that are installed with the OS X version of Python (under /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/), or keep them in the search path? I recommend hiding them, so that the newest versions of these packages will automatically be installed in user-accessible locations when needed. If you don't hide this directory, then you will occasionally get messages that pip was unable to remove an existing package in order to upgrade it to a later version (needed by a different package you are installing). In that case, you will need to run pip install --ignore-installed <package>, which will install the newer version and hide the system-installed version. However, if you hide the whole /System/.../Extras/... directory, you will lose access to some Apple packages that are not available via pip, i.e., CoreGraphics and bonjour. (If you need these, you may be able to get access by symlinking them into your site-packages directory.)

Now, here are the workarounds. These would be good practice on all versions of OS X, to avoid accidentally replacing or removing Python packages used by the operating system; however they are essential if you want to use user-installed packages with the Apple-supplied version of Python on OS X El Capitan (10.11).

Install pip

You probably did this already, but if not, you can use the following command to install pip for all users:

sudo -H easy_install pip
# pip script will be installed in /usr/local/bin

Or use this command to install pip for your own user account only:

easy_install --user pip
# pip script will be installed in ~/Library/Python/2.7/bin

Manage Shared File Locations

If you are installing packages for all users, create a file called .pydistutils.cfg with these lines (from https://github.com/pypa/pip/issues/426):

[install]
install-data=/usr/local
install-headers=/usr/local

If you usually use sudo -H pip ..., then you should put this file in /var/root (home directory for the root user). If you usually use sudo pip ..., then you should put this file in your own home directory (~).

These settings will prevent pip from trying to write shared items like headers and manpages under /Library/System. (The command at the top of this answer is a quicker version of the same thing.) These settings are needed because darwin-specific code in /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/install.py fails to set these variables to root-writeable locations (although it sets other variables correctly). There is more information on this at https://github.com/pypa/pip/issues/3177 .

If you install packages for your own user account only, shared items will automatically be installed under ~/Library/Python/2.7/. But you should add the following lines to your ~/.profile so the shared items will be found when you need them:

export PATH=~/Library/Python/2.7/bin:$PATH
export MANPATH=~/Library/Python/2.7/share/man:$MANPATH

Note: you will need to start a new shell or run these on the command line for the changes to take effect. You may also want to run hash -r if you've recently removed old scripts from the path.

Manage Python Path

You will need to ensure that the packages you install are higher in Python's search order than system-installed packages. The easiest way to do this is with .pth files. This follows @Sacrilicious's suggestion elsewhere on this page, but ensures that your user site-packages directory is searched before your system-wide site-packages directory, and both are searched before the standard library and Apple's Extras directory (both under /System/...). It also omits /System/.../Extras from the search path if desired.

Create a file called fix_mac_path.pth, with the text below. If you are installing packages for all users, fix_mac_path.pth should be placed in /Library/Python/2.7/site-packages. If you are installing only for your own user, fix_mac_path.pth should be in ~/Library/Python/2.7/lib/python/site-packages. (This file can have any name you want, but it has to be placed in one or both of these locations, and it has to end with .pth; also, all the text in this file has to be on one line.)

If you want to hide the Apple-installed packages in /System/.../Extras:

First run one of the following commands to get a working copy of pip/setuptools independent from the Apple-supplied version:

pip install --ignore-installed --user setuptools   # your account only
# or
sudo -H pip install --ignore-installed setuptools  # all users

Then put the following code in fix_mac_path.pth at the location specified above:

import sys; std_paths=[p for p in sys.path if p.startswith('/System/') and not '/Extras/' in p]; sys.path=[p for p in sys.path if not p.startswith('/System/')]+std_paths

If you want to keep using the Apple-installed packages, you don't need to install another copy of setuptools. Just put the following code in fix_mac_path.pth at the location specified above:

import sys; std_paths=[p for p in sys.path if p.startswith('/System/')]; sys.path=[p for p in sys.path if not p.startswith('/System/')]+std_paths

After this, you can use python -m site to make sure the path search order makes sense.

Install Packages

After this, you should be able to install new packages using one of the following commands.

For all users:

sudo -H pip install <package>

For your own user:

pip install --user <package>
  • Very clear directions with alternatives spelled out. I followed this guide and successfully was able to get past the "six" library problem, allowing me to install mitmproxy. – Paul Chernoch Jun 18 '16 at 14:35
  • 1
    This is really one of the most informative, complete, and helpful answers I have ever encountered on stackoverflow. Well done, and thanks. – cmsjr Jul 30 '16 at 8:37
  • @cmsjr, thanks! It took me a while to figure out what was going on, and this gives me a good place to refer back to it! – Matthias Fripp Aug 5 '16 at 22:57
  • 4
    By far the clearest and most complete explanation I've seen of this area, particularly cleanup of existing system Python mess - thank you! My main improvement would be more strongly recommend using a brew-installed Python, as this is very easy and provides a clean setup that doesn't require sudo. This is in addition to virtualenvs for development projects. – RichVel Aug 20 '16 at 7:48
  • What a disaster. God bless you. – Matt M. Jan 7 '17 at 7:11
31
+100

First, you're not disabling SIP as a way to address the issue. Sorry that's the reason that this folder creation is failing, but we have to work around it. Second, you lose all the niceties Apple thought they were giving you, like a bridge via pyObjC, when you install your own Python.
(I'll admit, they say you should install your own to /usr/local if you're a developer, I just object to doing it the easy way with brew.)

Logically, you'd think to install it somewhere SIP isn't blocking you, and this is almost certainly some dependency pip is resolving for you. Telling pip where you want it to perform installs isn't the remedy as much as having it ignore the pre-existing stuff in /System, which is where it would check for dependencies for many common installs that need, e.g.six, and SIP causes pip to break itself when it goes to try and upgrade it. It's actually pip's default behavior to install to /Library/Python/2.7/site-packages, you just need to say --ignore-installed to force it to install any updated versions of dependencies there.
...Unfortunately, you may then go to import this module in an interactive session or script and get nice fat failures - it's still looking at /System's un-upgraded package!

Since early setuptools days, Python has a way to make that lookup explicit. It's a bit odd-looking, (and should elicit pause to those of us who are security-conscious and not thrilled about the bluntness of shoving it to the top of the list without checking, but) you can place a file that ends with .pth (e.g. 'elcap.pth') in /Library/Python/2.7/site-packages to push that directory to the front of the lookup order with the following contents:
import sys; sys.path = ['/Library/Python/2.7/site-packages'] + sys.path

Start a new session, and a quick trip to python -m site will confirm that you've inserted that path into the first slot, and importing modules should work.

Oh, and after all this, try just installing with pip's --user option, or using a virtualenv - that's best practice for most folks anyway.

  • 1
    Now this is elegant and a good counterpoint to my simplistic "abandon the system Python and brew it" solution. If you need quick and dirty, this might not be worth the effort, but keeping pyObjC and one python has great merit. – bmike Oct 9 '15 at 2:37
  • 2
    More pythonic and sensible would be to install as --user or use a virtualenv. I'm just of the mind, as a sysadmin, that you install once for a system, and that the user should be able to override system. The objections to this solution may be that the system would be looking at an altered path, but there is the precedent that easy_install can write a .pth file as well. – Sacrilicious Oct 9 '15 at 4:25
  • 2
    I don't know if the brew way is the easy/wrong way – it's the matter of solving problems, and brew is solving my problems just spectacularly - thinking alone about how much effort it would take to play with the new toys (ruby 2.2.3, python3.5, MongoDB, Node4.2.1) in an convinient way, like typing a command and it recompiles, updates and takes the thrash out – well, so I take the easy path then :) I don't know what's the shipped OS X version of Python is so different aside the Cocoa and Threads support and a lot of packages I never used before and I indeed compiled my pile of Xcode/Kernel stuff – PJJ Oct 9 '15 at 5:48
  • I added elcap.pth to /Library/Python/2.7/site-packages, now I see that path listed at the 0 and 2 indices of sys.path. Then I tried sudo pip install --ignore-installed ipython and it runs until "Running setup.py install for pexpect", at which point pip still tries to upgrade something in /System/Library/Frameworks/Python.framework/Versions/2.7/share, and errors out. – kojiro Feb 28 '16 at 13:47
  • Installing virtualenv requires using pip, so that does not really solve any problems related to pip usage. As it is, I am installing it with pip --user and virtualenv is still not working. – user5359531 Aug 16 '16 at 17:55
4

I think its because of the SIP or System Integrity Protection, a kind of real-time file protection that feels like a windows anti-virus :) and stops any changes you want to make in the OS X system folders. Apple simply decided to disable system modifications – to be sure you are fully protected from the evil of internet and other computer related evil forces SIP is protecting us from.

If you want to disable SIP you need to boot into Recovery HD by holding Command + R keys simultaneously as you boot your Mac.

Open the terminal from the Utilities menu, and type in csrutil disable, then press return. Go to the Apple Menu to reboot.

SIP would be disabled from now on. I disabled it long time ago and haven't noticed anything wrong, some processes seems to complain but they always do on OS X so i'm not sure if it's caused by the lack of System Integrity 'protection'

My tip is to entirely get rid of SIP and/or use homebrew for all of your opensource/development software. Homebrew uses the /usr/local directory for it's installment and is not colliding with other system components, and homebrew builds are more up-to-date then the os x apple builds.

  • 1
    It makes no sense to down vote this answer. But, maybe someone is just cranky? +1 from me... – bmike Oct 6 '15 at 23:22
  • 1
    Thanks mate! Tried to put some humour, but not every one must be aligned with my understanding of it ;) – PJJ Oct 6 '15 at 23:52
  • Hmm - after reading sacrilicious' answer, I'm not so sure this is SIP. Thoughts? – bmike Oct 9 '15 at 2:51
  • 1
    This is indeed SIP preventing the creation of a folder(possibly for something like a man page) as a subdirectory of /System; off-limits as of 10.11. We can use csrutil to temporarily disable it if we're in a pinch, although I know that might feel like it could turn into whack-a-mole, but this doesn't seem like a good reason to employ it. Leaving it off is… why we can't have nice things. – Sacrilicious Oct 9 '15 at 4:31
  • I got it off all the time: statusSystem Integrity Protection status: enabled (Custom Configuration). Configuration: Apple Internal: disabled Kext Signing: disabled Filesystem Protections: disabled Debugging Restrictions: disabled DTrace Restrictions: disabled NVRAM Protections: disabled and don't have a reason to enable it since i got my system pretty tight closed and customised – PJJ Oct 9 '15 at 5:34
4

Used pip3 install <package> instead and solved the permission problem in pip.

4

Believe me, you don't really want the library to write anything at that path.

It was previously not recommended, but possible to write into /System/Library/Frameworks/Python.framework/Versions/2.7/ , but now it's not supported due to Apple SIP and so it's the library owner problem. The package distribution should be upgraded to work correctly with this update. Most packages were updated and install their content to /Library/Python/2.7/site-packages, but some packages were not updated.

For my case, it was greenlet library trying to write its .h file into System Frameworks folder:

How to fix it: sudo -H pip install greenlet --install-option "--install-headers=/Library/Python/2.7/lib/python/includes/" then sudo -H pip install gevent

For numpy, the fix is sudo -H pip install --ignore-installed -U numpy.

For other libraries, fixes vary from https://github.com/pypa/pip/issues/3177 to pip install --ignore-installed six and pip install --user (last one installs everything into /User//Library/Python/2.7/ path). See also current top answer to this post: https://apple.stackexchange.com/a/210021/169157

If you type python -m site it should include sys.path = [ ... '/Library/Python/2.7/site-packages', ... ] before system paths -- that's why (and how) it works.

3

I did it as follows:

brew install python

then after python is installed:

sudo easy_install pip
  • 3
    brew install python already installs pip with it. If you then install pip via easy_install, you likely wind up with two installations of pip that may lead to confusing problems. – NSSynapse May 29 '16 at 12:47
3

A very cool solution for this issue is using virtualenv (virtualenvwrapper), after you create a new environment for your project, you can use pip without issues, so I used virtualenvwrapper and these two lines fix the problem:

mkproject <project_name>
pip install <package_name>
  • How do you recommend installing virtualenv? pip install virtualenv or is pip not functioning and you need to resort to sudo easy_install +1 for what's here already. Cheers – bmike Jan 14 '16 at 19:24
  • 1
    pip is not working due permissions issue, so sudo easy_install works – neosergio Jan 14 '16 at 21:42
2

I have python3 installed on my mac while the original python2.7 comes with the OSX. So whenever I want to invoke the python3, i just type $python3 .... Maybe you should try $python3 get-pip.py, this works for me when I have almost the same problem with you.

2

Summary

I hit this problem on Mac OS X 10.11.6 (which has SIP) because I installed pip using the System easy_install, and the local easy-install.pth referred to system libraries.
When installing subsequent packages with pip some of the dependencies were resolved to the older System Python libraries.
The solution was to remove my locally installed libraries and reinstall a local python version (which includes pip) from https://www.python.org/downloads/ so that the System and locally installed python instances are kept separate.
I had tried using the fix_mac_path.pth fix in the answer by @mfripp however I found removing and reinstalling cleaner.

Cause

(Don't follow these steps)

I hit this problem whilst trying to install Ansible. I followed the Ansible docs for installation on OS X via pip
First I installed pip with sudo easy_install pip
This used the System easy_install at /usr/bin/easy_install and installed pip at /Library/Python/2.7/site-packages/pip
I got warnings as follows when installing pip but I ignored them and ploughed blindly on.

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
warning: no previously-included files found matching '.coveragerc'
...
Adding pip 9.0.1 to easy-install.pth file
Installing pip script to /usr/local/bin
...
Installed /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg
...
Finished processing dependencies for pip

Later looking at /Library/Python/2.7/site-packages/easy-install.pth, it looked like this

import sys; sys.__plen = len(sys.path)
./pip-9.0.1-py2.7.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

Then I installed ansible via pip

sudo pip install ansible

I got messages to say that requirements were already fulfilled by system libraries

Requirement already satisfied: setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from ansible)  
Requirement already satisfied: six>=1.4.1 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from cryptography>=1.1->paramiko->ansible)

Then when running ansible I got this problem

VersionConflict: (setuptools 1.1.6 (/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python), Requirement.parse('setuptools>=11.3'))

and a pip check revealed paramiko required a newer setuptools

pip check paramiko
matplotlib 1.3.1 requires tornado, which is not installed.
matplotlib 1.3.1 requires nose, which is not installed.
cryptography 1.7.2 has requirement setuptools>=11.3, but you have setuptools 1.1.6.

Note that ansible only required setuptools (no version) and so pip reported the dependency satisfied by the System setuptools.

Solution

I resolved it by uninstalling any local python libraries using the process hinted at https://docs.python.org/2.7/using/mac.html#getting-and-installing-macpython
For me this involved

sudo rm -rf /Library/Python

Then I removed the symlinks and executables in /usr/local/bin such as

sudo rm /usr/local/bin/ansible  # executable
sudo rm /usr/local/bin/python*  # symlinks to /Library/Python/2.7
sudo rm /usr/local/bin/easy_install*

and so on. I also removed any applications

sudo rm -rf /Applications/Python\ 2.7/

I then downloaded the 2.7.13 installer package for Mac OS X from https://www.python.org/downloads/ and installed it.

This installed a local python and pip at /Library/Frameworks/Python.framework/Versions/2.7 and symlinks in /usr/local/bin which is separate from the system libraries at /System/Library/Frameworks/Python.framework and /usr/bin so that I get

which pip
/Library/Frameworks/Python.framework/Versions/2.7/bin/pip
which ansible
/Library/Frameworks/Python.framework/Versions/2.7/bin/ansible

and in pip list

setuptools (28.8.0)
six (1.10.0)

ansible now works for me

1

For me, the PATH had gone missing for pip, this was confirmed by running python -m pip

Two potential solutions here, add pip back to PATH. In my case pip was out of date anyways, so upgrading fixed it:

python -m pip install --upgrade pip

✗ pip --version pip 9.0.1 from /Library/Python/2.7/site-packages (python 2.7)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .