Add llms.txt and llms-full.txt generation

[AdrianAtZyte]

To do:

• Wait until Provide more scrapy/sphinx-scrapy#1 is merged and released, then update the corresponding 4 files.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.82%. Comparing base (dc72308) to head (4629565).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #188      +/-   ##
==========================================
- Coverage   94.02%   93.82%   -0.21%     
==========================================
  Files          28       28              
  Lines        1958     1958              
==========================================
- Hits         1841     1837       -4     
- Misses        117      121       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.