Limit parameter not respected across various endpoints

[bradfordlynch]

Describe the bug
Endpoints such as /v3/trades and /v3/quotes support a limit parameter which according to the documentation "Limit the number of results returned, default is 1000 and max is 50000." However, this appears to be operating more like a page size than a limit on the number of results returned. At the very least the documentation should be updated for clarity.

To Reproduce
Run the following code:

quotes = []
for t in pg_client.list_quotes(
	ticker="AAPL",
	order="asc",
	limit=10,
	sort="timestamp",
	):
    quotes.append(t)

    if len(quotes) > 100:
        break

print(len(quotes))
# Result 101

Expected behavior
Based on the documentation, I would have expected the length of quotes to be 10 not 101. Note that this is basically the example from the docs and if you run it it will run for a very long time as it pages through the quotes for AAPL.

[justinpolygon]

Hey @bradfordlynch, you're probably seeing this https://github.com/polygon-io/client-python?tab=readme-ov-file#pagination-behavior. By default the client will auto paginate for you. The reason is that many of the APIs return tons of data and the client will automatically follow the next_url and stich the result together for you. If you wanted to turn that off there is a flag you can see like this client = RESTClient(api_key="<API_KEY>", pagination=False) on your client and it'll respect the limits.

[bradfordlynch]

Thanks, yup, I get that but the documentation--which features the Python client library--does not provide this color (I inferred that limit was actually a page size based on the behavior of the client). For example, see here: https://polygon.io/docs/rest/stocks/trades-quotes/quotes

The docs specifically say that limit is "Limit the number of results returned, default is 1000 and max is 50000." This seems confusing to me but it is your product.

[justinpolygon]

Yeah, that makes sense @bradfordlynch. I'll see if maybe we can alter the examples in the documentation to better show what's going on. Thanks for the added context on going from docs -> library. I 100% see what you're talking about.

[bradfordlynch]

I'm sure you don't want to edit the actual API spec itself. For me, if the docs had said something like "Limit the number of results returned per page, default is 1000 and max is 50000." Or anything to highlight that this is a page size limit rather than a hard limit on the number of results returned then I would not have been confused.

[KarmicTrades]

I had the same issue earlier this year, which sparked the implementation of the pagination flag update. I think it's important to note the API spec is for the endpoints, but the library's pagination functions are technically something different. The library passes most of the parameters straight through to the endpoint, but "limit" isn't passed with them.

Personally, I would like to see the parameters listed in the API documentation to be passed straight through to the endpoints, even when using the pagination functions (list_*) of the library. Any pagination-specific parameters could be named something different. If the "result limit" parameter for the endpoint is passed through, then the number of pages returned would be limited by the number of results requested. Without it, the pagination will continue until all available results have been received. This might result in 100 pages and 10,000 results, when only 500 results are needed and can easily be retrieved in 1-5 pages.

[justinpolygon]

Hi @bradfordlynch and @KarmicTrades, thank you both for the detailed reports and excellent feedback. We added a pagination flag a while back specifically to give full control over this behavior (sorry I forgot to update the ticket earlier).

Pagination is enabled by default (pagination=True) and the the client automatically fetches all pages, yielding results until none remain.

This fetches all TSLA trades, 100 per page:

client = RESTClient(api_key="<API_KEY>")
trades = [t for t in client.list_trades(ticker="TSLA", limit=100)]

To return a fixed number of results and stop, disable pagination you can use the pagination option. This returns at most 100 total trades, no additional pages:

client = RESTClient(api_key="<API_KEY>", pagination=False)
trades = [t for t in client.list_trades(ticker="TSLA", limit=100)]

I've also updated the main README to try and make this crystal clear going forward. Thanks again for raising this and for the great suggestions, this kind of feedback directly improves the library. I’m going to close this issue now as resolved, but please feel free to reopen or comment if anything else comes up!