M1 mac: mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e') - python

Question

When I try to run "scrapy startproject" in vscode, below error comes up.

Traceback (most recent call last):
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/bin/scrapy", line 5, in <module>
    from scrapy.cmdline import execute
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/__init__.py", line 12, in <module>
    from scrapy.spiders import Spider
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/spiders/__init__.py", line 11, in <module>
    from scrapy.http import Request
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/http/__init__.py", line 11, in <module>
    from scrapy.http.request.form import FormRequest
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/http/request/form.py", line 10, in <module>
    import lxml.html
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/lxml/html/__init__.py", line 53, in <module>
    from .. import etree
ImportError: dlopen(/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/lxml/etree.cpython-39-darwin.so, 0x0002): tried: '/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/lxml/etree.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/etree.cpython-39-darwin.so' (no such file), '/usr/lib/etree.cpython-39-darwin.so' (no such file)

And then I figure out that, the "mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')"is the main promblem. I tried with https://github.com/nmwsharp/robust-laplacians-py/issues/7 and I tried to uninstall the old packages and install it again using arch -arm64 pip install Scrapy. But it didn't work...

So...how to fix this problem...?

Answer 1

I use conda from miniforge for python packages etc. This works.

This means that I don't know first hand how pypi works, I just have looked at the documentation. I think by default it looks for prebuilt wheels and if it can't find one it builds the extension locally from source. This is no problem for pure python modules. However if the wheel should contain C code that has to be compiled then I think pypi has a large bug here in that it just looks for macos wheels and does not check the CPU architecture.

The error you have is that lxml has been built as an Intel binary. Looking at the downloads for lxml at pypi there are no wheels for Apple arm CPUs.

So if not conda miniforge you need to compile lxml yourself or force pip to do the build locally. pypi documentation suggests

You can override pip`s default behavior by e.g. using its –no-binary option.

I would also note that Macports provides arm versions of python and lxml and possibly scrapy

Answer 2

What it worked for me was:

pip uninstall lxml
ARCHFLAGS="-arch arm64" pip install lxml --compile --no-cache-dir

Answer 3

I've faced with the same issue (and created this ticket with details of my research). But it ended with removing the installed package:

pip uninstall lxml

... and installing it building from sources:

pip install --no-binary lxml lxml 

Note the useful recommendation about platform specifics (for MacOS, for example).

Answer 4

This is a case of some python libraries not being compiled for M1 Apple Silicon CPUs, but only for Intel.

Try running VSCode in Rosetta.

Answer 5

I got this when running an npm command that used compiled C packages. Solution was to run terminal in x86_64 mode and using node v14.x.x, as newer versions of node didn't work.

Answer 6

Running the commands below resolved the problem for me:

• arch -arm64e gem install nokogiri -v '1.13.6' --platform=ruby -- --use-system-libraries
• bundle

Answer 7

I got this same import error with uamqp library

ImportError: dlopen(/Users/maheshwaran/Documents/newenv/lib/python3.10/site-packages/uamqp/c_uamqp.cpython-310-darwin.so, 0x0002): tried: '/Users/maheshwaran/Documents/newenv/lib/python3.10/site-packages/uamqp/c_uamqp.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))

I resolved this import error by running below command:

1. Uninstalling uamqp library:

   pip uninstall uamqp

2. Setting ARCHFLAGS for arm64 architecture and using no cache so it will download respective library suits arm64 architecture.

   ARCHFLAGS="-arch arm64" pip install uamqp --compile --no-cache-dir

Hope this helps you as well !!