Document Uploads: Fix checkout_timeout error

[akanshaaa19]

Is your feature request related to a problem?
We are experiencing a checkout_timeout error during document uploads. This issue is preventing the successful upload of essential files, which impacts functionality for users.

Describe the solution you'd like

• Investigate the cause of the checkout_timeout error.
• Implement a fix to ensure reliable document uploads.
• Monitor for any recurring issues after the fix is applied.

Message

** (Glific.ThirdParty.Kaapi.Error) Kaapi document upload failed for, filename=AH1063_Newsletter.pdf reason: :checkout_timeout organization_id: 205 - antara foundation

View on AppSignal: https://appsignal.com/project-tech4dev/sites/5f480c425ac13f7330101f30/exceptions/incidents/468?id=last

[AkhileshNegi]

High Level Summary

• Tanu attempting to upload 265 files to assistant, experiencing failures
• Current system uploads one file at a time through documents endpoint which creates a copy of files in S3 and once documents are uploaded collection endpoint is called which then downloads the copy of files and upload it to OpenAI and create vector store
• Multiple timeout errors occurring:
  • 49 checkout timeout errors for Andhra project
  • Errors clustered within 1-minute window (1:55-1:56 PM)
  • Same :checkout_timeout seen in Gemini API calls
• Root cause likely connection pool exhaustion from simultaneous requests so not might not be originating from Kaapi API side as similar issue is seen in Gemini calls as well

Proposed Technical Solutions

• Batch size optimization:
  • Current default batch size = 1 (processes files individually)
  • Recommend Glific to use batch size of 15 for large file uploads
  • Experiment planned to test 265 files with higher batch sizes locally by Nishika

User Experience Workarounds
Based on few tries of uploading files 3 times for this assistant
https://akshita.glific.com/assistants/784
recommendations for now will be

• Upload files in batches of 6-8 at a time
• Wait for files to get uploaded
• Click Save only after all files uploaded
• Click on save button and wait for A new version is being created
• After that follow same process

[AkhileshNegi]

Traced the issue to how our HTTP client stack is configured. Glific uses Tesla with the Hackney adapter for outbound HTTP calls. Hackney manages requests using a connection pool, which by default allows ~50 concurrent connections. When ~200 files were uploaded almost simultaneously, the first batch occupied the available connections and the rest had to wait in the queue.

In Hackney’s code, when a request tries to acquire a connection it calls the pool with a checkout timeout (which defaults to the 8 seconds). If a connection isn’t available within that time, Hackney returns {:error, :checkout_timeout}.

This GitHub issue explains about this pool exhaustion but mentions that it was addressed around Hackney 1.17, and Glific is already on 1.20.1, so the issue we saw isn’t the old bug resurfacing. The more likely cause is simply pool saturation due to high concurrency — uploads take longer than normal API calls and keep connections busy, so when ~200 requests start together the queue builds up and some requests hit the checkout timeout.

In short: this was not a bug in the upload logic, but a connection pool limit being hit under heavy parallel uploads.

Possible action items:
• Set a limit on maximum allowed file numbers while selecting
• Configure a larger or dedicated Hackney pool (latter one is the best option)
• Evaluate moving Tesla to Finch adapter in the future for better concurrency behaviour or move to Req which is a more modern HTTP client that also uses Finch under the hood

[AkhileshNegi]

We experimented with creating a vector store using a larger number of files and a batch size of 10, and observed a significant reduction in processing time. For instance, creating a vector store with 50 files took about 2 minutes with this batch size.

We’ve created a PR on our end to set the default batch size to 10. From your end, please ensure that in the request body being sent to the Create Collection API, if the batch_size parameter is included, its value is not set to 1. You can either set it to 10 (or higher) or omit the parameter entirely, in which case the default value of 10 will be applied automatically.

#689