Separate out the CRONs service away from the UOP core

[YufanKambang]

This discussion is regarding a possible idea, we have currently had two meeting one internally with submission in ICD, and one with ESS (Jekabs.K, Yoganandan.P). Both parties so far are interested in this idea.

Currently we have all CRONs jobs stored in the source code of UOP (user-offfice-project)/UOC (user-office-core). This we believe can be improved, as it is currently residing as a monolith.
There is a archived repo User-office-crons (UOCrons): https://github.com/UserOfficeProject/user-office-cron

We want to abstract this out of UOP/UOC and make it importable and then configurable.
Of which we want to be similar to user-office-factory:
https://github.com/UserOfficeProject/user-office-factory

These jobs may call some backend endpoint, connect directly to the database, we want to be able to spy on the jobs.

Ideas

• We can bring out a background worker architecture by using rabbit-mq
• Consider reactivating old repo and keep code in one place.
• Merge things together, than separate, what jobs each collaborator has
• Make it generic jobs for all the facilities
• Have the facilities themselves set up custom job for their region of UOP
• How are we using templates?

Possible problem

• When the contributor of UOP each want a CRONs service for there logic and the CRONs jobs are scheduled. How do we allow different/identical scheduled CRONs jobs to happen synchronously within one UOP?
• We have a Kubernetes cluster running multiple of the same job, this job however sends an event to users when completed. How are we going to make sure that a only the significant amount of post-job events occur within a schedule period?

[gnyiri]

I think that we should let facilities set up custom jobs dynamically. Ideally, the CRON service would be a generic scheduler that will start jobs at given frequencies. A new job can be plugged in to the service by implementing an abstract job interface, something like this:

interface Job {
getFrequency(): int;
init(): void;
run(): void;
teardDown();
}

Some jobs can be pre-defined and stored in the core repo while others will be implemented on site and not made public.

[YufanKambang]

We have review so far three possible solution internally in ICD, these are:

Library + Dedicated Runner Service

This model provides a shared Node.js/TypeScript CRONs library and a dedicated runner service that collaborators run in their own environment.

This will compose:

• Central team that publishes NPM package
  The shared library provides the job logic and scheduling framework
• Collaborators that run these jobs
  Each collaborator owns their runtime, configuration, and secrets

Core artefacts:

• NPM package
  This will contain and expose all common CRON jobs
  Provide a schedular setup, using a Node CRON library
  Allow job configuration with params and env variables
• Base runner service
  Simple Node app that loads the shared packages, and reads configurations for jobs and schedule
  Possibly packaged as a docker image

Deployment options:

• Use the provided Docker runner image
• Integrate the library into a custom Node app

Implementation:

Collaborators

• Choose a deployment style based on deployment
• Install or reference the artefact
• Select and configure jobs
• Run the runner service

Central team

• Implement and maintain shared jobs and scheduling logic in the NPM package.
• Publish new versions and update the base runner image.
• Provide documentation on configuration and job semantics

[YufanKambang]

Celery

This is a python based distributed task queue with a built-in periodic scheduler called Celery Beat, it is widely used in Django/Flask ecosystems to run scheduled background jobs
In our context, the main codebase is TypeScript/Node
This is do not have a high understanding of but I think it is very interesting and should be consider

Core concepts:

• A Celery Beat process acting as the centralized periodic scheduler
• One or more Celery worker processes that execute the CRON jobs
• A message broker like mailChimp or RabbitMQ and typically a result backend for job state

Collaborators

• Run their own Celery Beat instance
• Run their own Celery workers connected to a broker they control

Central team

• Celery would run as a separate Python service alongside our Node/TS services
• To run our TypeScript jobs, Celery would
  Call HTTP endpoints on our Node services
  or
  Trigger a shell command that runs a TS job
• Provide the collaborators with the common CRON jobs, this would require to work with both Python and Node/TS
  Collaborators with their own Python/Celery stack could use Celery Beat to orchestrate calls into our Node/TS APIs

Implementation

To note: Celery itself does not run Node/TS code directly
Maybe happens like this:

1. Celery Beat schedules tasks
2. Workers make HTTP calls into our Node/TS services, or invoke a CLI that runs the TS job

[YufanKambang]

Kubernetes CRONjob

this is a Kubernetes-native approach to scheduled tasks using CRONjobs
A Kubernetes CRONjob

• Registers a CRON-style schedule
• On each schedule tick, the CRONjob controller creates a job object
• Each job then creates one or more pods that run to completion

Deployment

Cluster & namespaces

• We run our workloads in Kubernetes clusters
• Each product/team/tenant gets its own namespace; all of that tenant’s CRONjobs live there

Configuration & delivery

• CRONjobs are defined as standard Kubernetes resources (YAML) and live in the application’s Git repo
• Deployment can use the same CI/CD pipeline as services
  Define how envs are structured
  Provide CRONjob “scaffolding”
• Environment-specific differences (e.g. schedule frequency, resource limits, secrets references) can be handled via:
  Overlay mechanisms (e.g. kustomize/Helm) or separate per-environment directories

Implementation

• Design the common CRON jobs
  Clarify behavior like idempotency, expected duration and data access needs
• Build the containerized task
  The container should take configuration via env vars and/or config files and emit structured logs for cluster logging
• Define the CRONjobs
  Image and tag for the job container, resource requests/limits.
  I think volumes/secrets for configuration and credentials
  Possibly ServiceAccount and RBAC bindings

Collaborator will do

• Job design & implementation
• with the base image they build on top of it and/or configure it

[YufanKambang]

Minutes from meeting with Yoga.

Yufan presented a K8S solution using namespaces and Kubernetes native CronJobs resource and to allow cronjobs to be extracted out and placed into a separate repository.
Yoga identified that this is solution while very good for facilities, may be unnecessary to start implementing right away as it suggests to use a separate repository to hold all the configs and jobs. The cronjobs in user-office-core at the moment are localised for that service and set up to use the configuration in the repo. So if we moved the jobs we would need to do a lot more configuration within this new repository so that the cronjob moved can work in the service they reside in. If we are worried about collaborator running unnecessary jobs, they are able to add a config layer that dictates which jobs to not run.

The problems that we have with cron jobs is we do not have control over these cronjobs; such as no knowing how many times these cron jobs are run, audits such as how long they have, how the order of the cron jobs should be queued to prevent contrary clashing jobs occurring synchronously.

Moving forward is to start another discussion to talk about how to resolves these problems that we have identified and agreed to be most impactful.