33

When I try to run "scrapy startproject" in vscode, below error comes up.

Traceback (most recent call last):
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/bin/scrapy", line 5, in <module>
    from scrapy.cmdline import execute
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/__init__.py", line 12, in <module>
    from scrapy.spiders import Spider
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/spiders/__init__.py", line 11, in <module>
    from scrapy.http import Request
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/http/__init__.py", line 11, in <module>
    from scrapy.http.request.form import FormRequest
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/scrapy/http/request/form.py", line 10, in <module>
    import lxml.html
  File "/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/lxml/html/__init__.py", line 53, in <module>
    from .. import etree
ImportError: dlopen(/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/lxml/etree.cpython-39-darwin.so, 0x0002): tried: '/Users/admin/Desktop/development/project/scrapyTutorial/venv/lib/python3.9/site-packages/lxml/etree.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/etree.cpython-39-darwin.so' (no such file), '/usr/lib/etree.cpython-39-darwin.so' (no such file)

And then I figure out that, the "mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')"is the main promblem. I tried with https://github.com/nmwsharp/robust-laplacians-py/issues/7 and I tried to uninstall the old packages and install it again using arch -arm64 pip install Scrapy. But it didn't work...

So...how to fix this problem...?

0

8 Answers 8

12

I use conda from miniforge for python packages etc. This works.

This means that I don't know first hand how pypi works, I just have looked at the documentation. I think by default it looks for prebuilt wheels and if it can't find one it builds the extension locally from source. This is no problem for pure python modules. However if the wheel should contain C code that has to be compiled then I think pypi has a large bug here in that it just looks for macos wheels and does not check the CPU architecture.

The error you have is that lxml has been built as an Intel binary. Looking at the downloads for lxml at pypi there are no wheels for Apple arm CPUs.

So if not conda miniforge you need to compile lxml yourself or force pip to do the build locally. pypi documentation suggests

You can override pip`s default behavior by e.g. using its –no-binary option.

I would also note that Macports provides arm versions of python and lxml and possibly scrapy

5
  • I have been tried...but the same error comes back. It seems that lxml library does not working some reason...
    – Ryan
    Feb 12, 2022 at 6:39
  • Heyyyyyyyy. I just figured out the solution!!!!!!!!!!!!! First I really appreciate @ mmmmmm help. I run conda install -c conda-forge lxml, then the scrapy startproject works!!!!!
    – Ryan
    Feb 12, 2022 at 6:55
  • But Conda is not free for commercial usage. A paid license is needed, isn't it ?
    – Jeff T.
    Apr 8, 2022 at 6:54
  • I use Macports and it works perfectly. Thanks @mmmmmm.
    – Jeff T.
    Apr 8, 2022 at 10:31
  • 2
    @JeffT. Looking at docs.conda.io/projects/conda/en/latest It says it is open source. ALso github.com/conda/conda says it is BSD .Anaconda does have pricing
    – mmmmmm
    Apr 8, 2022 at 10:43
13

What it worked for me was:

pip uninstall lxml
ARCHFLAGS="-arch arm64" pip install lxml --compile --no-cache-dir
2
  • This should be the answer May 11, 2023 at 20:43
  • 1
    Would be interesting to know if ARCHFLAGS="-arch arm64 -arch arm64e -arch x86_64" creates a universal binary. May 15, 2023 at 20:43
6

I've faced with the same issue (and created this ticket with details of my research). But it ended with removing the installed package:

pip uninstall lxml

... and installing it building from sources:

pip install --no-binary lxml lxml 

Note the useful recommendation about platform specifics (for MacOS, for example).

4

This is a case of some python libraries not being compiled for M1 Apple Silicon CPUs, but only for Intel.

Try running VSCode in Rosetta.

2
  • Thanks for your help. I really appreciated. But, I tried run scrapy startproject VSCode rosetta terminal...still it shows same error...
    – Ryan
    Feb 11, 2022 at 11:56
  • I tried that without success. With uname in the terminal I get x86_64, but doing ´platform.uname()` in Python still get arm64.
    – Christian
    Jan 19, 2023 at 17:25
0
  1. download arm64 .whl from https://pypi.org/project/lxml/#files

  2. pip3 install xxx.whl

-1

I got this when running an npm command that used compiled C packages. Solution was to run terminal in x86_64 mode and using node v14.x.x, as newer versions of node didn't work.

1
-1

Running the commands below resolved the problem for me:

  • arch -arm64e gem install nokogiri -v '1.13.6' --platform=ruby -- --use-system-libraries
  • bundle
1
  • 2
    How has a ruby gem anything to do with a python application?
    – mmmmmm
    May 16, 2022 at 16:29
-1

I got this same import error with uamqp library

ImportError: dlopen(/Users/maheshwaran/Documents/newenv/lib/python3.10/site-packages/uamqp/c_uamqp.cpython-310-darwin.so, 0x0002): tried: '/Users/maheshwaran/Documents/newenv/lib/python3.10/site-packages/uamqp/c_uamqp.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))

I resolved this import error by running below command:

  1. Uninstalling uamqp library:

    pip uninstall uamqp

  2. Setting ARCHFLAGS for arm64 architecture and using no cache so it will download respective library suits arm64 architecture.

    ARCHFLAGS="-arch arm64" pip install uamqp --compile --no-cache-dir

Hope this helps you as well !!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .