Skip to content

so-okada/bXiv

BranchesTags

Repository files navigation

Application Info

bXiv delivers arXiv daily new submissions, abstracts, cross-lists, and replacements by bluesky posts, replies, and reposts. We use python3 scripts with atproto. bXiv is not affiliated with arXiv.

Setup

  • Install atproto, pandas, ratelimit, nameparser, feedparser, and beautifulsoup4.

     % pip3 install atproto pandas ratelimit nameparser feedparser beautifulsoup4

  • Let bXiv.py be executable.

     % chmod +x bXiv.py

  • Put the following python scripts in the same directory.

    • bXiv.py
    • bXiv_post.py
    • bXiv_format.py
    • bXiv_daily_feed.py
    • arXiv_feed_parser.py
    • bXiv_variables.py

  • Configure switches.json, logfiles.json, and aliases.json in the tests directory for your settings.

    • accesses.json specifies bluesky access keys and whether to use new submissions/abstracts/cross-lists/replacements by bXiv.

    • logfiles.json indicates log file locations for post summaries, posts, reposts, unreposts, and replies. You can check their formats by mathACb_post_summaries.csv, mathACb_posts.csv, mathACb_reposts.csv, mathACb_unreposts.csv, mathACb_replies.csv, and mathACb_quotes.csv in the tests/logfiles directory. bXiv needs a post log file for cross-lists and replacements. Other log files are useful to avoid duplication errors of posts and reposts.

    • aliases.json tells bXiv aliases of arXiv category names. For example, math.IT is an alias of cs.IT. Without this file, bXiv of rss feeds returns no new submissions, when you take the category name math.IT. If provided, bXiv replaces category names by their aliases for new submissions, cross-lists, and replacements.

  • Configure bXiv_variables.py for your settings.

    • bXiv_variables.py assigns format parameters for bXiv posts and access frequencies for arXiv and bluesky.

Notes

  • arXiv_feed_parser.py is a simple arXiv feed parser for bXiv. We use this via bXiv_daily_feed.py to regularly obtain data.

  • Outputs of bXiv can differ from arXiv new submission web pages. First, this can be due to bugs in my scripts or connection errors. Second, items of arXiv rss feeds are not necessarily the same as those of arXiv new submission web pages (see https://mastoxiv.page/@vela/109829294232368163). Third, arXiv_feed_parser for an arXiv category C gives new submissions whose primary subjects are the category C. Then, bXiv for the category C counts and posts a new paper whose principal subject matches the category C.

    • For example, let us assume that there is no new paper whose principal subject matches the category C, but there is a new paper P whose non-principal subject matches the category C. Then, the arXiv new submission web page of the category C lists the paper P as a new submission of the category C, not as a cross-list. However, bXiv keeps considering the paper P as a cross-list for the category C. Then, the output of bXiv for the category C differs from the arXiv new submission web page of the category C.

    • So, bXiv puts one single post of a new paper across bots in the access keys. If configured, bXiv tries to repost and quote for cross-lists and replacements from bots in the access keys.

  • On the use of metadata of arXiv articles, the web page Terms of Use for arXiv APIs says that "You are free to use descriptive metadata about arXiv e-prints under the terms of the Creative Commons Universal (CC0 1.0) Public Domain Declaration." and "Descriptive metadata includes information for discovery and identification purposes, and includes fields such as title, abstract, authors, identifiers, and classification terms."

Usage

% ./bXiv.py -h
usage: bXiv.py [-h] --switches_keys SWITCHES_KEYS
                [--logfiles LOGFILES] [--aliases ALIASES]
                [--captions CAPTIONS] [--mode {0,1}]

arXiv daily new submissions by posts, abstracts by
replies, cross-lists by reposts, and replacements by
quotes and reposts.

optional arguments:
  -h, --help            show this help message and exit
  --switches_keys SWITCHES_KEYS, -s SWITCHES_KEYS
                        output switches and api keys in
                        json
  --logfiles LOGFILES, -l LOGFILES
                        log file names in json
  --aliases ALIASES, -a ALIASES
                        aliases of arXiv categories in
                        json
  --captions CAPTIONS, -c CAPTIONS
                        captions of arXiv categories in
                        json
  --mode {0,1}, -m {0,1}
                        1 for twitter and 0 for stdout
                        only

Sample stdouts

  • New submissions for math.AC and math.AG with no log files:

     % ./bXiv.py -s tests/switches.json -m 1
 **process started at xxxx-xx-xx xx:xx:xx (UTC)
 starting a thread of retrieval/new submissions/abstracts for math.AC
 getting daily entries for math.AC
 waiting for a next thread of retrieval/new submissions/abstracts
 new submissions for math.AC
 no log files
 no log files

 utc: xxxx-xx-xx xx:xx:xx 
 thread arXiv category: math.AC
 arXiv id: 
 url: https://bsky.app/profile/
 post method: post
 post mode: 1
 url: https://bsky.app/profile/xxxxxxxxxxxxxxxxxxxxxxxx
 text: [xxxx-xx-xx Sun (UTC), 2 new articles found for mathAC]

 starting a thread of retrieval/new submissions/abstracts for math.AG
 getting daily entries for math.AG
 joining threads of retrieval/new submissions/abstracts

 utc: xxxx-xx-xx xx:xx:xx
 thread arXiv category: math.AC
 arXiv id: xxxx.xxxxx
 url: https://bsky.app/profile/
 post method: post
 post mode: 1
 url: https://bsky.app/profile/xxxxxxxxxxxxxxxxxxxxxxxx
 text: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

 ....

 **process ended at xxxx-xx-xx xx:xx:xx (UTC)
 **elapsed time from the start: xx:xx:xx

  • New submissions, abstracts, cross-lists and replacements (if any) for math.AC and math.AG:

     % ./bXiv.py -s tests/switches.json -l tests/logfiles.json -a tests/aliases.json -c tests/captions.json -m 1
 **process started at xxxx-xx-xx xx:xx:xx (UTC)
 starting a thread of retrieval/new submissions/abstracts for math.AC
 getting daily entries for math.AC
 waiting for a next thread of retrieval/new submissions/abstracts
 new submissions for math.AC

 utc: xxxx-xx-xx xx:xx:xx 
 thread arXiv category: math.AC
 arXiv id: 
 url: https://bsky.app/profile/
 post method: post
 post mode: 1
 url: https://bsky.app/profile/xxxxxxxxxxxxxxxxxxxxxxxxx
 text: [xxxx-xx-xx Sun (UTC), 2 new articles found for mathAC Commutative Algebra]

 starting a thread of retrieval/new submissions/abstracts for math.AG
 getting daily entries for math.AG
 joining threads of retrieval/new submissions/abstracts

 utc: xxxx-xx-xx xx:xx:xx
 thread arXiv category: math.AC
 arXiv id: xx07.xxxxxx
 url: https://bsky.app/profile/
 post method: post
 post mode: 1
 url: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

 new submissions for math.AG

 utc: xxxx-xx-xx xx:xx:xx
 thread arXiv category: math.AG
 arXiv id: 
 url: https://bsky.app/profile/
 post method: post
 post mode: 1
 url: https://bsky.app/profile/xxxxxxxxxxxxxxxxxxxxxxxxxxxx
 text: [xxxx-xx-xx Sun (UTC), 4 new articles found for mathAG Algebraic Geometry]

 utc: xxxx-xx-xx xx:xx:xx
 thread arXiv category: math.AC
 arXiv id: xx07.xxxxxx
 url: https://bsky.app/profile/xxxxxxxxxxxxxxxxxxxx
 post method: reply
 post mode: 1
 url: https://bsky.app/profile/xxxxxxxxxxxxxxxxxxxx
 text:  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [1/4 of https://arxiv.org/abs/xx07.xxxxxv1]

 .....
 
 **crosslist process started at xxxx-xx-xx xx:xx:xx (UTC) 
 **elapsed time from the start: xx:xx:xx
 start a crosslist thread of math.AC
 waiting for a next crosslist thread
 start a crosslist thread of math.AG
 joining crosslist threads

 **replacement process started at xxxx-xx-xx xx:xx:xx (UTC)
 **elapsed time from the start: xx:xx:xx
 **elapsed time from the cross-list start: xx:xx:xx
 quote-replacement starts
 start a quote-replacement thread of math.AC
 waiting for a next quote-replacement thread
 start a quote-replacement thread of math.AG
 
 .....
 
 utc: xxxx-xx-xx xx:xx:xx
 thread arXiv category: math.AG
 arXiv id: xx10.xxxx
 url: https://bsky.app/profile/xxxxxxxxxxxxxxxxxxxx
 post method: quote
 post mode: 0
 url: https://bsky.app/profile/
 text: This https://arxiv.org/abs/xxxx.xxxx has been replaced.....

 .....

 repost-replacement starts
 start a repost-replacement thread of math.AC
 waiting for a next quote-replacement thread
 start a repost-replacement thread of math.AG
 
 utc: xxxx-xx-xx xx:xx:xx 
 thread arXiv category: math.AG
 arXiv id: xx00.xxxxx
 url: https://bsky.app/profile/xxxxxxxxxxxxxxxxxxxx
 post method: unrepost
 post mode: 1
 url: https://bsky.app/profile/xxxxxxxxxxxxxxxxxxxx
 text: 

 utc: xxxx-xx-xx xx:xx:xx 
 thread arXiv category: math.AG
 arXiv id: xx00.xxxxx
 url: https://bsky.app/profile/xxxxxxxxxxxxxxxxxxxx
 post method: repost
 post mode: 1
 url: https://bsky.app/profile/xxxxxxxxxxxxxxxxxxxx
 text: 

 .....

 **process ended at xxxx-xx-xx xx:xx:xx (UTC)
 **elapsed time from the start: xx:xx:xx
 **elapsed time from the cross-list start: xx:xx:xx
 **elapsed time from the replacement start: xx:xx:xx

  • Without the option -c tests/captions.jsonabove, you get

     text: [xxxx-xx-xx Sun (UTC), 4 new articles found for mathAG]

    instead of

     text: [xxxx-xx-xx Sun (UTC), 4 new articles found for mathAG Algebraic Geometry]

Versions

  • 0.0.1, initial release, 2025-01-12.

  • 0.0.2, minor updates, 2025-02-04.

  • 0.0.3, an update for createAccount endpoint ratelimit (100 per 5 min), 2025-02-18.

  • 0.0.4, a fix for unrepost/repost, 2025-03-02.

  • 0.0.5, a fix for name handling, 2026-02-18.

  • 0.0.6, a fix for feedparser timeout, 2026-04-19.

List of Bots

bXiv scripts for the bots are running on the OCI free tier (as of late March 2026).

Author

So Okada, so.okada@gmail.com, https://so-okada.github.io/

Motivation

This is an open-science practice (see https://github.com/so-okada/twXiv#motivation). Since 2013-04, the author has been running twitter bots for all arXiv math categories. Since 2023-01, the author has been running mastodon bots for all arXiv categories with toXiv. Since 2025-02, the author has been running the bluesky bots for all arXiv categories (except stat.AP, stat.CO, and cs.NI as explained in List of Bots) with bXiv.

License

AGPLv3

About

bXiv delivers arXiv daily new submissions, abstracts, cross-lists, and replacements by bluesky posts, repliles, and reposts. We use python3 scripts with atproto. bXiv is not affiliated with arXiv.

github.com/so-okada/bXiv

Topics

python3 arxiv-daily atproto bluesky-bot cross-listing

Resources

Readme

License

AGPL-3.0 license
Activity

Stars

1 star

Watchers

2 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors 2

Languages