MINIFICPP-2669 - Reduce controller service api#2065
MINIFICPP-2669 - Reduce controller service api#2065adamdebreceni wants to merge 11 commits intoapache:mainfrom
Conversation
36ff46b to
b61e730
Compare
fd945e9 to
6b53a92
Compare
| .artifact = "minifi-system", | ||
| .group = "org.apache.nifi.minifi", | ||
| .type = "org.apache.nifi.minifi.core.RecordSetReader", | ||
| .version = "1.0.0" |
There was a problem hiding this comment.
I think the version of components shipped with minifi should always match the minifi version, what do you think?
There was a problem hiding this comment.
changed it to use the minifi version
| * Controller Service base class that contains some pure virtual methods. | ||
| * | ||
| * Design: OnEnable is executed when the controller service is being enabled. | ||
| * Note that keeping state here must be protected in this function. |
There was a problem hiding this comment.
I don't understand this sentence, could you elaborate or rephrase?
There was a problem hiding this comment.
outdated, updated it
| } | ||
| if (thread_manager_->isAboveMax(current_workers_)) { | ||
| auto max = thread_manager_->getMaxConcurrentTasks(); | ||
| auto max = 1; // thread_manager_->getMaxConcurrentTasks(); |
There was a problem hiding this comment.
why did this change? leftover debug thing?
There was a problem hiding this comment.
removed the whole thread management logic/controller service
a1401a3 to
8f3c38b
Compare
There was a problem hiding this comment.
Pull request overview
Refactors MiNiFi C++ controller service APIs to reduce/reshape the exposed surface area, introducing a new “API/implementation” split and updating call sites across core, extensions, and tests.
Changes:
- Introduces new controller-service API types (API, context, descriptor, factory, metadata) and reworks controller service registration/lookup.
- Replaces many direct
ControllerServiceImplusages withControllerServiceBase+ControllerServiceInterfaceand wraps implementations in a newcore::controller::ControllerService. - Updates thread pool/controller service wiring and adjusts numerous unit/integration tests to the new controller service access patterns.
Reviewed changes
Copilot reviewed 118 out of 118 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| minifi-api/include/minifi-cpp/core/controller/ControllerServiceProvider.h | Removed legacy API header for controller service provider. |
| minifi-api/include/minifi-cpp/core/controller/ControllerServiceMetadata.h | Replaced old node/service API types with a minimal metadata struct. |
| minifi-api/include/minifi-cpp/core/controller/ControllerServiceInterface.h | Reduced controller service interface to a marker-style interface. |
| minifi-api/include/minifi-cpp/core/controller/ControllerServiceFactory.h | Added factory interface for creating controller service API implementations. |
| minifi-api/include/minifi-cpp/core/controller/ControllerServiceDescriptor.h | Added descriptor API for declaring supported properties. |
| minifi-api/include/minifi-cpp/core/controller/ControllerServiceContext.h | Added context API for property access during enable. |
| minifi-api/include/minifi-cpp/core/controller/ControllerServiceApi.h | Added new controller-service API surface (initialize/onEnable/notifyStop). |
| minifi-api/include/minifi-cpp/core/ProcessContext.h | Switched controller service return type to ControllerServiceInterface. |
| minifi-api/include/minifi-cpp/core/ControllerServiceApiDefinition.h | Added version field to controller service API definition. |
| minifi-api/include/minifi-cpp/core/ClassLoader.h | Added overload to register controller service factories. |
| minifi-api/include/minifi-cpp/controllers/ThreadManagementService.h | Migrated to ControllerServiceInterface. |
| minifi-api/include/minifi-cpp/controllers/SSLContextServiceInterface.h | Migrated to ControllerServiceInterface and added API version. |
| minifi-api/include/minifi-cpp/controllers/RecordSetWriter.h | Migrated to ControllerServiceInterface and added API version. |
| minifi-api/include/minifi-cpp/controllers/RecordSetReader.h | Migrated to ControllerServiceInterface and added API version. |
| minifi-api/include/minifi-cpp/controllers/AttributeProviderService.h | Migrated to ControllerServiceInterface. |
| libminifi/test/unit/YamlFlowSerializerTests.cpp | Updated controller service accessors to new return types. |
| libminifi/test/unit/UpdatePolicyTests.cpp | Updated controller service creation and access to implementation. |
| libminifi/test/unit/ProcessorConfigUtilsTests.cpp | Updated controller-service parsing tests for new wrappers/interfaces. |
| libminifi/test/unit/NetworkPrioritizerServiceTests.cpp | Updated tests to use wrapped controller services and implementation accessors. |
| libminifi/test/unit/NetUtilsTest.cpp | Updated SSL context service retrieval and usage. |
| libminifi/test/unit/JsonFlowSerializerTests.cpp | Updated controller service accessors to new return types. |
| libminifi/test/unit/ComponentManifestTests.cpp | Updated example controller service type hierarchy for new APIs. |
| libminifi/test/libtest/unit/MockClasses.h | Updated mock controller service usage for new ControllerServiceInterface. |
| libminifi/test/libtest/unit/ControllerServiceUtils.h | Added test helper for constructing wrapped controller services. |
| libminifi/test/keyvalue-tests/VolatileMapStateStorageTest.cpp | Updated controller service implementation retrieval helper usage. |
| libminifi/test/keyvalue-tests/PersistentStateStorageTest.cpp | Updated controller service implementation retrieval helper usage. |
| libminifi/test/integration/ControllerServiceIntegrationTests.cpp | Updated controller service implementation retrieval helper usage. |
| libminifi/test/integration/C2ControllerEnableFailureTest.cpp | Updated dummy controller service to new base/interface split. |
| libminifi/src/core/state/nodes/ResponseNodeLoader.cpp | Updated UpdatePolicy controller retrieval to new implementation access path. |
| libminifi/src/core/logging/LoggerConfiguration.cpp | Switched SSLContextService creation to createAndEnable. |
| libminifi/src/core/controller/ControllerServiceProvider.cpp | Renamed provider impl usage to new provider type. |
| libminifi/src/core/controller/ControllerServiceNode.cpp | Updated node implementation accessors and signatures. |
| libminifi/src/core/ClassLoader.cpp | Added wrapper to adapt controller service factories into object factories. |
| libminifi/src/controllers/UpdatePolicyControllerService.cpp | Removed now-unneeded legacy lifecycle methods. |
| libminifi/src/controllers/SSLContextService.cpp | Added createAndEnable helper using new wrapper service type. |
| libminifi/src/controllers/NetworkPrioritizerService.cpp | Updated prioritizer token handling and linkage with new API model. |
| libminifi/src/c2/protocols/RESTSender.cpp | Switched SSLContextService creation to createAndEnable. |
| libminifi/src/c2/ControllerSocketProtocol.cpp | Switched SSLContextService creation to createAndEnable. |
| libminifi/src/c2/C2Agent.cpp | Updated UpdatePolicy controller retrieval to new implementation access path. |
| libminifi/src/RemoteProcessGroupPort.cpp | Updated controller service lookup return type and SSL fallback creation. |
| libminifi/src/FlowController.cpp | Updated thread pool controller service provider wiring. |
| libminifi/include/io/NetworkPrioritizer.h | Removed global prioritizer factory and API method from interface. |
| libminifi/include/core/controller/StandardControllerServiceProvider.h | Updated to new provider base type and base include. |
| libminifi/include/core/controller/StandardControllerServiceNode.h | Updated to new node base type and includes. |
| libminifi/include/core/controller/ForwardingControllerServiceProvider.h | Updated to new provider base and removed deprecated methods. |
| libminifi/include/core/controller/ControllerServiceProvider.h | Consolidated provider API/impl and removed linked-service state helpers. |
| libminifi/include/core/controller/ControllerServiceNodeMap.h | Updated include paths for new controller service node header. |
| libminifi/include/core/controller/ControllerServiceNode.h | Replaced old node interface with new concrete base and templated impl accessor. |
| libminifi/include/core/controller/ControllerServiceLookup.h | Removed enabled/enabling/name helpers; kept lookup methods only. |
| libminifi/include/core/controller/ControllerService.h | Added new wrapper controller service type around ControllerServiceApi. |
| libminifi/include/core/ProcessGroup.h | Updated include path for controller service node. |
| libminifi/include/core/ProcessContextImpl.h | Updated to return ControllerServiceInterface and use templated node accessor. |
| libminifi/include/core/FlowConfiguration.h | Updated include path for controller service node. |
| libminifi/include/controllers/UpdatePolicyControllerService.h | Migrated to ControllerServiceBase + ControllerServiceInterface. |
| libminifi/include/controllers/ThreadManagementService.h | Migrated to ControllerServiceBase and updated initialization. |
| libminifi/include/controllers/SSLContextService.h | Migrated to ControllerServiceBase and added createAndEnable. |
| libminifi/include/controllers/NetworkPrioritizerService.h | Migrated to ControllerServiceBase + ControllerServiceInterface and embedded prioritizer state. |
| libminifi/include/c2/HeartbeatReporter.h | Updated include path for controller service provider. |
| libminifi/include/c2/ControllerSocketProtocol.h | Updated include path for controller service provider. |
| libminifi/include/c2/C2Protocol.h | Updated include path for controller service provider. |
| libminifi/include/SchedulingAgent.h | Updated include path for controller service provider. |
| libminifi/include/FlowController.h | Updated include path for controller service provider. |
| extensions/standard-processors/tests/unit/XMLRecordSetWriterTests.cpp | Updated tests to use wrapped controller services and implementation accessors. |
| extensions/standard-processors/tests/unit/XMLReaderTests.cpp | Updated tests to use wrapped controller services and implementation accessors. |
| extensions/standard-processors/tests/unit/TailFileTests.cpp | Updated test controller service mock to match new interface expectations. |
| extensions/standard-processors/tests/unit/JsonRecordTests.cpp | Updated tests to use wrapped controller services and implementation accessors. |
| extensions/standard-processors/tests/unit/ControllerServiceTests.cpp | Updated tests to construct new wrapped controller service type. |
| extensions/standard-processors/controllers/XMLRecordSetWriter.h | Updated service construction style and removed legacy common macros. |
| extensions/standard-processors/controllers/XMLReader.h | Updated service construction style and removed legacy common macros. |
| extensions/standard-processors/controllers/VolatileMapStateStorage.h | Updated construction style and removed legacy common macros. |
| extensions/standard-processors/controllers/VolatileMapStateStorage.cpp | Updated base initialization call to ControllerServiceBase. |
| extensions/standard-processors/controllers/PersistentMapStateStorage.h | Updated construction style and removed legacy common macros. |
| extensions/standard-processors/controllers/PersistentMapStateStorage.cpp | Updated base initialization call to ControllerServiceBase. |
| extensions/standard-processors/controllers/JsonTreeReader.h | Updated construction style and removed legacy common macros. |
| extensions/standard-processors/controllers/JsonRecordSetWriter.h | Updated construction style and removed legacy common macros. |
| extensions/sql/tests/mocks/MockODBCService.h | Updated construction style and removed legacy common macros. |
| extensions/sql/services/ODBCConnector.h | Updated ODBCService construction style and removed legacy common macros. |
| extensions/sql/services/DatabaseService.h | Migrated to ControllerServiceBase + ControllerServiceInterface. |
| extensions/sql/services/DatabaseService.cpp | Updated base initialization call to ControllerServiceBase. |
| extensions/smb/tests/utils/MockSmbConnectionControllerService.h | Updated test mock to remove legacy controller-service macros. |
| extensions/smb/tests/SmbConnectionControllerServiceTests.cpp | Updated to use templated node implementation accessor. |
| extensions/smb/tests/PutSmbTests.cpp | Updated to use templated node implementation accessor. |
| extensions/smb/tests/ListSmbTests.cpp | Updated to use templated node implementation accessor. |
| extensions/smb/tests/ListAndFetchSmbTests.cpp | Updated to use templated node implementation accessor. |
| extensions/smb/tests/FetchSmbTests.cpp | Updated to use templated node implementation accessor. |
| extensions/smb/SmbConnectionControllerService.h | Migrated to ControllerServiceBase + ControllerServiceInterface. |
| extensions/rocksdb-repos/controllers/RocksDbStateStorage.h | Updated construction style and removed legacy common macros. |
| extensions/rocksdb-repos/controllers/RocksDbStateStorage.cpp | Updated base initialization call to ControllerServiceBase. |
| extensions/kubernetes/controllerservice/KubernetesControllerService.h | Updated construction style and removed legacy common macros. |
| extensions/kubernetes/controllerservice/KubernetesControllerService.cpp | Updated base initialization call to ControllerServiceBase. |
| extensions/gcp/tests/GCPCredentialsControllerServiceTests.cpp | Updated to use templated node implementation accessor. |
| extensions/gcp/controllerservices/GCPCredentialsControllerService.h | Migrated to ControllerServiceBase + ControllerServiceInterface. |
| extensions/elasticsearch/ElasticsearchCredentialsControllerService.h | Migrated to ControllerServiceBase + ControllerServiceInterface. |
| extensions/couchbase/tests/PutCouchbaseKeyTests.cpp | Updated to use templated node implementation accessor. |
| extensions/couchbase/tests/MockCouchbaseClusterService.h | Updated test mock to remove legacy controller-service macros. |
| extensions/couchbase/tests/GetCouchbaseKeyTests.cpp | Updated to use templated node implementation accessor. |
| extensions/couchbase/controllerservices/CouchbaseClusterService.h | Migrated to ControllerServiceBase + ControllerServiceInterface. |
| extensions/civetweb/tests/ListenHTTPTests.cpp | Switched SSLContextService creation to createAndEnable. |
| extensions/azure/processors/AzureStorageProcessorBase.cpp | Updated controller service lookup return type. |
| extensions/azure/controllerservices/AzureStorageCredentialsService.h | Migrated to ControllerServiceBase + ControllerServiceInterface. |
| extensions/aws/tests/MultipartUploadStateStorageTest.cpp | Updated state storage construction to use metadata-based constructor. |
| extensions/aws/tests/AWSCredentialsServiceTest.cpp | Updated to use templated node implementation accessor. |
| extensions/aws/controllerservices/AWSCredentialsService.h | Migrated to ControllerServiceBase + ControllerServiceInterface. |
| extension-framework/src/controllers/keyvalue/KeyValueStateStorage.cpp | Removed legacy constructor in favor of new base. |
| extension-framework/include/utils/ProcessorConfigUtils.h | Updated controller service parsing to use ControllerServiceInterface. |
| extension-framework/include/controllers/keyvalue/KeyValueStateStorage.h | Migrated to ControllerServiceBase + ControllerServiceInterface. |
| extension-framework/include/controllers/RecordSetWriter.h | Migrated to ControllerServiceBase + interface exposure. |
| extension-framework/include/controllers/RecordSetReader.h | Migrated to ControllerServiceBase + interface exposure. |
| extension-framework/include/controllers/AttributeProviderService.h | Migrated to ControllerServiceBase + interface exposure. |
| encrypt-config/FlowConfigEncryptor.cpp | Updated controller service implementation retrieval call site. |
| core-framework/src/utils/ThreadPool.cpp | Updated controller service provider API and thread manager lookup. |
| core-framework/include/utils/ThreadPool.h | Updated provider API from lookup pointer to function callback. |
| core-framework/include/core/controller/ControllerServiceFactoryImpl.h | Added templated factory implementation for controller services. |
| core-framework/include/core/controller/ControllerServiceBase.h | Added new base implementation for controller service APIs. |
| core-framework/include/core/controller/ControllerService.h | Removed legacy ControllerServiceImpl header. |
| core-framework/include/core/Resource.h | Updated resource registration to also register controller services via factories. |
| controller/tests/ControllerTests.cpp | Switched SSLContextService creation to createAndEnable. |
| controller/MiNiFiController.cpp | Switched SSLContextService creation to createAndEnable. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
minifi-api/include/minifi-cpp/core/controller/ControllerServiceContext.h
Show resolved
Hide resolved
8f3c38b to
d6bf004
Compare
Closes #2065 Signed-off-by: Martin Zink <martinzink@apache.org>
4074e73 to
0972302
Compare
Closes #2065 Signed-off-by: Martin Zink <martinzink@apache.org>
Co-authored-by: Martin Zink <martin.zink@protonmail.com>
0972302 to
1628e00
Compare
|
|
||
| namespace org::apache::nifi::minifi::core::controller { | ||
|
|
||
| // A base class that helps with controller service development, contains common functionalities. |
There was a problem hiding this comment.
I don't think this comment is useful, the name of the class implies all of this.
There was a problem hiding this comment.
I disagree here. There is room for improvement in the comment, but a base could be a base for many reasons, and here it clarifies that it's a base that partially implements the API. Code reuse through inheritance is a bad pattern, but at least the comment makes it clear that this is happening here.
There was a problem hiding this comment.
I think in this form this comment has no added value, the first part of the comment is just the reiteration of the class name, ControllerServiceBase implies that this is a base class and it is used for controller services. The second part could have added value if it was improved as you said, but looking at the interface of the class the comment seems to be trivial. I think it should be either removed or improved as you suggested.
| /** | ||
| * Function is called when Controller Services are enabled and being run | ||
| */ |
There was a problem hiding this comment.
This also seems redundant
| /** | ||
| * Controller Service base class that contains some pure virtual methods. | ||
| * | ||
| * Design: OnEnable is executed when the controller service is being enabled. | ||
| * Note that keeping state here must be protected in this function. | ||
| */ |
There was a problem hiding this comment.
This comment is not valid anymore. For this interface it would be beneficial to explain what the purpose of this class is and how is it used.
There was a problem hiding this comment.
I agree here. It's confusing to have ControllerServiceApi, ControllerServiceInterface, and ControllerServiceBase. At least the first two mean the same thing in my understanding, so some clarification would be nice, ideally by brainstorming together about a better name that inherently communicates the purpose and the differences, but alternatively by writing them in a comment or developer documentation.
| virtual void initialize(ControllerServiceDescriptor& descriptor) = 0; | ||
| virtual void onEnable(ControllerServiceContext& context, const std::shared_ptr<Configure>& configuration, const std::vector<std::shared_ptr<ControllerServiceInterface>>& linked_services) = 0; | ||
| virtual void notifyStop() = 0; | ||
| virtual ControllerServiceInterface* getControllerServiceInterface() = 0; |
There was a problem hiding this comment.
I see the reason behind this, but it is a bit convoluted. It should get a comment explaining it and how to implement it.
| namespace org::apache::nifi::minifi::core::controller { | ||
|
|
||
| // A base class that helps with controller service development, contains common functionalities. | ||
| class ControllerServiceBase : public ControllerServiceApi { |
There was a problem hiding this comment.
I think I'd generally call these kinds of classes AbstractThing rather than ThingBase, and the ThingApi could be just Thing, but we can have more discussions and brainstorming around the naming patterns.
| /** | ||
| * Controller Service base class that contains some pure virtual methods. | ||
| * | ||
| * Design: OnEnable is executed when the controller service is being enabled. | ||
| * Note that keeping state here must be protected in this function. | ||
| */ |
There was a problem hiding this comment.
I agree here. It's confusing to have ControllerServiceApi, ControllerServiceInterface, and ControllerServiceBase. At least the first two mean the same thing in my understanding, so some clarification would be nice, ideally by brainstorming together about a better name that inherently communicates the purpose and the differences, but alternatively by writing them in a comment or developer documentation.
| return false; | ||
| } | ||
|
|
||
| static constexpr auto ImplementsApis = std::array<ControllerServiceApiDefinition, 0>{}; |
There was a problem hiding this comment.
This is quite confusing: core::controller::ControllerServiceApi is the general API class, but we also have core::ControllerServiceApi which is a special thing needed to build the manifest, along with ControllerServiceApiDefinition, ProvidesApi and ImplementsApis. So the word "Api" is used in two completely different senses in lines 133 and 136.
We should rename the old classes, or move them to a more specific namespace having to do with manifests, or both.
There was a problem hiding this comment.
I've encountered and renamed this in https://github.com/apache/nifi-minifi-cpp/pull/2096/changes#diff-8c1d24228b38431bc9fef4283084f78f6f5c9493d47a04bf31963f5064974713L133
There was a problem hiding this comment.
great, thanks! I think I would rename ProvidesApi and ImplementsApis, too, so all four are in line, but I'll comment that on your pull request then :D
| bool supportsDynamicProperties() const final { | ||
| return false; | ||
| } |
There was a problem hiding this comment.
This used to return the value of the static SupportsDynamicServices, now it is a constant false. Why did this change?
If we want to remove this option (I don't think we have any controller services where SupportsDynamicProperties == true), should we remove the static field, as well?
| if (const auto is_default = getProperty(DefaultPrioritizer.name) | utils::andThen(parsing::parseBool)) { | ||
| if (*is_default) { | ||
| if (io::NetworkPrioritizerFactory::getInstance()->setPrioritizer(sharedFromThis<NetworkPrioritizerService>()) < 0) { | ||
| throw std::runtime_error("Can only have one prioritizer"); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
I'm not sure what this did before, but if we are not using the DefaultPrioritizer property any longer, then it should be either removed or deprecated and the doc string updated.
| auto processor_impl = std::make_unique<T>(core::controller::ControllerServiceMetadata{ | ||
| .uuid = uuid.value(), | ||
| .name = std::string{name}, | ||
| .logger = minifi::core::logging::LoggerFactory<T>::getLogger(uuid.value()) | ||
| }); | ||
| return std::make_unique<core::controller::ControllerService>(name, uuid.value(), std::move(processor_impl)); |
There was a problem hiding this comment.
typo:
| auto processor_impl = std::make_unique<T>(core::controller::ControllerServiceMetadata{ | |
| .uuid = uuid.value(), | |
| .name = std::string{name}, | |
| .logger = minifi::core::logging::LoggerFactory<T>::getLogger(uuid.value()) | |
| }); | |
| return std::make_unique<core::controller::ControllerService>(name, uuid.value(), std::move(processor_impl)); | |
| auto controller_service_impl = std::make_unique<T>(core::controller::ControllerServiceMetadata{ | |
| .uuid = uuid.value(), | |
| .name = std::string{name}, | |
| .logger = minifi::core::logging::LoggerFactory<T>::getLogger(uuid.value()) | |
| }); | |
| return std::make_unique<core::controller::ControllerService>(name, uuid.value(), std::move(controller_service_impl)); |
| const auto* const controller_service_node_after = process_group_after->findControllerService(controller_service_id); | ||
| REQUIRE(controller_service_node_after); | ||
| const auto* const controller_service_after = controller_service_node_before->getControllerServiceImplementation(); | ||
| const auto controller_service_after = controller_service_node_after->getControllerServiceImplementation(); |
There was a problem hiding this comment.
👍 (I think this was my bug)
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?
For code changes:
For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.