Skip to content

RestIngestor enhancements#2130

Draft
aestoltm wants to merge 12 commits intoubccr:mainfrom
aestoltm:cider_resource_specs
Draft

RestIngestor enhancements#2130
aestoltm wants to merge 12 commits intoubccr:mainfrom
aestoltm:cider_resource_specs

Conversation

@aestoltm
Copy link
Copy Markdown
Contributor

@aestoltm aestoltm commented Dec 3, 2025

Description

General changes to allow for the following:

  • RestIngestor methods will now be inherited when extended
  • HTTP headers can be set
  • Can parse REST response for nested fields. The structure for fields that are parsing a specific field look like: "db_column_name": [ "intermediate_field_i", "desired_field" ]

Changes allow current cider resource information to be ingested through ETLv2. Additionally to help with pulling project/grant info with RestIngestor

Motivation and Context

Small enhancements to RestIngestor.

Tests performed

Tested with development of CiDeR resource specs

Checklist:

  • The pull request description is suitable for a Changelog entry
  • The milestone is set correctly on the pull request
  • The appropriate labels have been added to the pull request

@aestoltm aestoltm self-assigned this Dec 3, 2025
@aestoltm aestoltm added enhancement Enhancement of the functionality of an existing feature Category:ETL Extract Transform Load php Pull requests that update Php code labels Dec 3, 2025
@aestoltm aestoltm changed the title Cider resource specs RestIngestor enhancements Mar 10, 2026
@aestoltm aestoltm marked this pull request as ready for review March 10, 2026 15:43
@aestoltm aestoltm added this to the 11.5.0 milestone Mar 10, 2026
} // verifyTransformDirective()

/* ------------------------------------------------------------------------------------------
* Parses PHP obj/array and retrieves desired field. Supports a wildcard, "?", for the last item in an array.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the use case for adding support for the 'last item in array' wildcard?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Originally was using this to get the latest conversion factors from the cider api (last element). However, the cider pipeline is going to create a history of the conversion factors so I do not actually use it anymore. I can't think of any other cases so we probably can remove it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is better to not add it for now. If we need a capability in future then should be easy to add it in based on this work.

@aestoltm aestoltm requested a review from jpwhite4 March 31, 2026 14:58
@aestoltm aestoltm marked this pull request as draft April 2, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Category:ETL Extract Transform Load enhancement Enhancement of the functionality of an existing feature php Pull requests that update Php code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants