[BUG] 小红书抓取失败

[zymshirley]

🔍 问题检查清单

🐛 问题描述

尝试了扫码和cookie进行抓取都是弹出浏览器，扫码登录后有登录态，但是没有下一步的操作
我尝试了不同账号、不同IP局无法抓取

📝 复现步骤

1. 扫码或cookie登录
2. 弹出浏览器页面
3. 自动关闭浏览器页面后报错

💻 运行环境

• 操作系统: mac
• Python版本: Python 3.9.6
• 是否使用IP代理: 代理和直连都尝试了
• 是否使用VPN翻墙软件：代理和直连都尝试了
• 目标平台(抖音/小红书/微博等): 小红书

📋 错误日志

[11:35:24]
[ERR]
raise DataFetchError(err_msg)
[11:35:24]
[ERR]
media_platform.xhs.exception.DataFetchError: {"code":-1,"success":false}
[11:35:24]
[DATA]
The above exception was the direct cause of the following exception:
[11:35:24]
[DATA]
Traceback (most recent call last):
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/main.py", line 157, in
[11:35:24]
[DATA]
run(main, async_cleanup, cleanup_timeout_seconds=15.0, on_first_interrupt=_force_stop)
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/tools/app_runner.py", line 109, in run
[11:35:24]
[DATA]
asyncio.run(_runner())
[11:35:24]
[DATA]
File "/Users/gm/.local/share/uv/python/cpython-3.11.14-macos-aarch64-none/lib/python3.11/asyncio/runners.py", line 190, in run
[11:35:24]
[DATA]
return runner.run(main)
[11:35:24]
[DATA]
^^^^^^^^^^^^^^^^
[11:35:24]
[DATA]
File "/Users/gm/.local/share/uv/python/cpython-3.11.14-macos-aarch64-none/lib/python3.11/asyncio/runners.py", line 118, in run
[11:35:24]
[DATA]
return self._loop.run_until_complete(task)
[11:35:24]
[DATA]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[11:35:24]
[DATA]
File "/Users/gm/.local/share/uv/python/cpython-3.11.14-macos-aarch64-none/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
[11:35:24]
[DATA]
return future.result()
[11:35:24]
[DATA]
^^^^^^^^^^^^^^^
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/tools/app_runner.py", line 96, in _runner
[11:35:24]
[DATA]
await app_main()
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/main.py", line 110, in main
[11:35:24]
[DATA]
await crawler.start()
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/media_platform/xhs/core.py", line 119, in start
[11:35:24]
[DATA]
await self.get_creators_and_notes()
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/media_platform/xhs/core.py", line 209, in get_creators_and_notes
[11:35:24]
[DATA]
all_notes_list = await self.xhs_client.get_all_notes_by_creator(
[11:35:24]
[DATA]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/media_platform/xhs/client.py", line 600, in get_all_notes_by_creator
[11:35:24]
[DATA]
notes_res = await self.get_notes_by_creator(
[11:35:24]
[DATA]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/media_platform/xhs/client.py", line 574, in get_notes_by_creator
[11:35:24]
[DATA]
return await self.get(uri, params)
[11:35:24]
[DATA]
^^^^^^^^^^^^^^^^^^^^^^^^^^^
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/media_platform/xhs/client.py", line 168, in get
[11:35:24]
[DATA]
return await self.request(
[11:35:24]
[DATA]
^^^^^^^^^^^^^^^^^^^
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
[11:35:24]
[DATA]
return await fn(*args, **kwargs)
[11:35:24]
[DATA]
^^^^^^^^^^^^^^^^^^^^^^^^^
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 47, in call
[11:35:24]
[DATA]
do = self.iter(retry_state=retry_state)
[11:35:24]
[DATA]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[11:35:24]
[DATA]
File "/Users/gm/Documents/Program/MediaCrawler/.venv/lib/python3.11/site-packages/tenacity/init.py", line 326, in iter
[11:35:24]
[DATA]
raise retry_exc from fut.exception()
[11:35:24]
[ERR]
tenacity.RetryError: RetryError[<Future at 0x117d23d10 state=finished raised DataFetchError>]
[11:35:24]
[WARN]
Crawler exited with code: 1

📷 错误截图

[dosubot]

Hi @zymshirley! I'm Dosu and I’m helping the MediaCrawler team.

根据你的错误日志，这是一个已知问题。错误 DataFetchError: {"code":-1,"success":false} 发生在 get_notes_by_creator 调用时，是由于小红书 /api/sns/web/v1/user_posted 端点的签名算法问题导致的，目前尚无有效修复方案。

如果你是使用 --type creator 模式抓取创作者笔记，这个功能暂时无法使用。

可以尝试的临时解决方案：

1. 如果还遇到登录问题，可参考 #852 的解决方法：

   • 确认 config/base_config.py 中 HEADLESS = False
   • 多次尝试扫码（2-3次）
   • 删除 browser_data 目录后重试

2. 切换爬取模式：如果只是想抓取笔记内容而不是特定创作者的所有笔记，可以尝试使用关键词搜索模式 --type search 或单个笔记详情模式 --type detail，这些端点的签名问题已被修复。

这个问题需要等待项目更新签名算法才能彻底解决。

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}