Fix package reporting implementation

[lrafeei]

This PR attempts to fix an issue where some modules are not loaded in the environment section of the UI (the modules do not get passed into the update_loaded_modules endpoint). It also fixes a bug where it does not reload the modules upon agent restart.

Previously the logic attempted to load as many modules as possible during a 2 second window by iterating through a plugins generator. However, it seems that some modules that have a longer loading time get missed.

The logic here has been changed to continue to load the next module if the previous module took less than 0.5 seconds and the amount of time uploading modules is less than 5.0 seconds. If the previous module takes longer than 0.5 seconds to load, the agent will wait until the next harvest cycle to resume.

[github-actions]

✅MegaLinter analysis: Success

Descriptor	Linter	Files	Fixed	Errors	Warnings	Elapsed time
✅ ACTION	actionlint	8		0	0	1.03s
✅ MARKDOWN	markdownlint	7	0	0	0	1.37s
✅ PYTHON	ruff	1030	0	0	0	1.15s
✅ PYTHON	ruff-format	1030	0	0	0	0.37s
✅ YAML	prettier	19	0	0	0	1.65s
✅ YAML	v8r	19		0	0	5.36s
✅ YAML	yamllint	19		0	0	0.75s

See detailed reports in MegaLinter artifacts

Show us your support by starring ⭐ the repository

[codecov-commenter]

Codecov Report

❌ Patch coverage is 50.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.96%. Comparing base (77df266) to head (467cc52).

Files with missing lines	Patch %	Lines
newrelic/core/application.py	50.00%	3 Missing and 3 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1670   +/-   ##
=======================================
  Coverage   81.95%   81.96%           
=======================================
  Files         214      214           
  Lines       25803    25794    -9     
  Branches     4082     4083    +1     
=======================================
- Hits        21148    21143    -5     
+ Misses       3263     3256    -7     
- Partials     1392     1395    +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hmstepanek]

A couple thoughts here-I'm not sure exactly why it would not be reporting the module(s) again but I had one thought. Seems unlikely to be the reason but I didn't see anything else that would cause that. You could add some log statements or a metric in or something to maybe help you debug what's going on.

[hmstepanek]

I wonder-can you just simplify this to be the following and leave MAX_PACKAGE_CAPTURE_TIME_PER_SLOW_HARVEST=2:

Suggested change

	while ((time.time() - stopwatch_start) < 0.5) and (
	while (time.time() - start) < MAX_PACKAGE_CAPTURE_TIME_PER_SLOW_HARVEST:

This way it will at least load 1 module before exiting. I realize this is slow harvest but I wonder if 5s is a bit long.

[hmstepanek]

This seems unlikely but I wonder if what's happening for the AI team who is testing this is they are never entering the while loop because of that less than .5 check? Are they seeing any modules reporting at all? Another way you could implement this that might be simpler and avoid this potential issue is:

start = time.time()
for module in self.plugins:
    self._active_session.send_loaded_modules([module])
    if time.time() - start > MAX_PACKAGE_CAPTURE_TIME_PER_SLOW_HARVEST:
        break
else:
    self._remaining_plugins = False

[hmstepanek]

I'm wondering if we really need the self._remaining_plugins or can we just set self.plugins = False?

[lrafeei]

Oh, like override the generator object to be a boolean when it runs out and use that as the conditional?

[hmstepanek]

logger.debug("Looping over plugins.")
while package in self.plugins:
   logger.debug(f"Recording package {package}")
   self._active_session.send_loaded_modules([next(package)])
   if (time.time() - start) < MAX_PACKAGE_CAPTURE_TIME_PER_SLOW_HARVEST
       logger.debug("Exiting plugins loop.)
       break

[hmstepanek]

Suggested change

	MAX_PACKAGE_CAPTURE_TIME_PER_SLOW_HARVEST = 3.0
	MAX_PACKAGE_CAPTURE_TIME_PER_SLOW_HARVEST = 2.0

[JiwaniZakir]

The new harvest loop in application.py around line 1291 has an inverted condition that effectively breaks after sending a single package every harvest cycle. The line if (time.time() - start) < MAX_PACKAGE_CAPTURE_TIME_PER_SLOW_HARVEST: break causes an immediate break because the first iteration almost never exceeds 3 seconds — the < should be >= to match the original behavior of stopping once the time budget is exhausted. This is a semantic inversion of the old while ... (elapsed < MAX) guard.

Additionally, the new code calls send_loaded_modules([package]) individually per package rather than batching, which introduces significantly more network overhead compared to the original approach of accumulating and sending once per cycle. If the intent was to stream packages incrementally, that trade-off should be documented.

Using next(self.plugins, False) with False as the sentinel is unconventional and could mask bugs if package itself is falsy for any reason — None is the standard sentinel for next().

The addition of self.plugins = plugins() on restart is correct and closes a gap where a restarted agent would have had an exhausted generator, but it would benefit from a comment explaining why the reset is scoped to the restart path rather than being part of a broader state reset.