Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# ArmoniK Docs

Docs for ArmoniK

## Installation

> Be aware to be at the root of the repository

```bash
python -m venv .venv-doc
```

Then activate the virtual environment:

```bash
source .venv-doc/bin/activate
```

And install dependencies:

```bash
pip install -r .docs/requirements.txt
```

## Usage

To build the docs locally, run the following command:

```bash
sphinx-build -M html .docs build
```

Outputs can be found in `build/html/index.html`.
3 changes: 3 additions & 0 deletions .docs/_static/custom.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.wy-nav-content {
max-width: 100% !important;
}
51 changes: 51 additions & 0 deletions .docs/conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information

project = "ArmoniK.Extensions.Java"
copyright = "2021-%Y, ANEO"
author = "ANEO"
release = "main"

# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

extensions = ["myst_parser", "sphinxcontrib.mermaid"]

templates_path = ["_templates"]
exclude_patterns = ["requirements.txt", "README.md"]
suppress_warnings = ["myst.header"]

# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

html_theme = "sphinx_rtd_theme"
html_static_path = ["_static"]
html_css_files = ["custom.css"]
html_search = True
html_extra_path = ["api/client/apidocs"]

# -- Options for source files ------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-source-files
source_suffix = {
".rst": "restructuredtext",
".txt": "markdown",
".md": "markdown",
}

# -- Options MyST Parser ------------------------------------------------
myst_fence_as_directive = ["mermaid"]
myst_heading_anchors = 3

# -- Options to show "Edit on GitHub" button ---------------------------------
html_context = {
"display_github": True, # Integrate GitHub
"github_user": "aneoconsulting", # Username
"github_repo": "ArmoniK.Extensions.Java", # Repo name
"github_version": "main", # Version
"conf_py_path": "/.docs/", # Path in the checkout to the docs root
}
47 changes: 47 additions & 0 deletions .docs/content/0.overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# ArmoniK Java SDK — Overview

## Client–Worker Model

ArmoniK implements a **Client–Worker** distributed processing model. Responsibilities are clearly separated:

**Client Responsibilities**
- Create and manage sessions.
- Submit tasks and define execution dependencies (task graphs).
- Manage input/output blobs and error handling.

**Worker Responsibilities**
- Implement the computation logic for each task.
- Read input blobs and produce output blobs.
- Optionally submit new tasks (dynamic workflows).

## SDK Goals

- **Simplify** the development of Clients and Workers.
- Provide **pre-built Worker containers** following unified conventions.
- Offer a **high-level Worker API** with consistent lifecycle and error handling.
- Provide **simplified Client APIs** to create sessions, submit tasks, and manage data.
- Promote **best practices**: idempotency, error reporting, observability.
- Ensure **compatibility and long-term maintainability** across ArmoniK versions.
- Ensure **interoperability**: clients and workers can be mixed across languages.

## Interoperability

The SDK is designed to enable cross-language interoperability — you can mix clients and workers across SDK languages.

**Example:** A Java client orchestrates tasks executed by Python or C# workers.

Two conventions make this possible:

**Convention 1 — Payload structure**

The payload is a JSON association table mapping logical names to blob IDs:
```json
{"inputs": {"A": "id1"}, "outputs": {"result": "id2"}}
```

**Convention 2 — Task options**

Dynamic library loading is enabled via standardized task options carrying:
- Path to the library
- Symbol (class/function name)
- Blob ID for the library artifact
136 changes: 136 additions & 0 deletions .docs/content/1.client.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
# ArmoniK Java SDK — Client

## Role of the Client SDK

The client SDK provides a language-friendly interface to interact with ArmoniK's control plane. It allows you to:

- Create and manage **sessions**.
- Define and submit **tasks**, optionally with dependencies.
- Create and manage **input and output blobs**.
- Monitor task progress and retrieve results.
- Use optional asynchronous callback mechanisms.
- Ensure consistent workflow semantics: retries, error propagation, idempotency.

---

## Entry Point: `ArmoniKClient`

The main entry point is `ArmoniKClient`, configured via `ArmoniKConfig` which describes:
- Endpoints
- TLS configuration
- Retry policy

Use try-with-resources for automatic cleanup:

```java
try (ArmoniKClient client = new ArmoniKClient(config)) {
// ...
}
```

**Available operations on `ArmoniKClient`:**
- Create a new `Session`
- Retrieve an existing `Session`
- Cancel a `Session`
- Close a `Session`

---

## Session Creation

Define a session with `SessionDefinition`, then submit it to get a `SessionHandle`.

`SessionDefinition` components:

| Component | Description |
|---|---|
| Partition IDs | Where tasks can execute |
| `TaskConfiguration` | Default configuration for all tasks |
| `BlobCompletionListener` | Callback when an output blob is `COMPLETED` or `ABORTED` |

---

## Blob Completion Listener

`BlobCompletionListener` is a callback interface for async result notifications. Register it in `SessionDefinition` to enable event-driven result handling.

```java
BlobCompletionListener listener = new BlobCompletionListener() {
@Override
public void onBlobSuccess(Blob blob) { /* blob is ready */ }

@Override
public void onBlobError(BlobError error) { /* handle error */ }
};
```

---

## Working with Blobs

| Type | Description |
|---|---|
| `InputBlobDefinition` | Data sent to workers |
| `OutputBlobDefinition` | Placeholder for expected results |
| `BlobHandle` | Reference to an already uploaded/downloaded blob |

Input data can come from:
- In-memory byte array
- File
- Custom source by implementing the `BlobData` interface

`SessionHandle` can also create session-scoped blobs directly.

---

## Task Submission

Create a `TaskDefinition` specifying:
- **Inputs**: `InputBlobDefinition` or `BlobHandle`
- **Outputs**: `OutputBlobDefinition`
- **Optional**: override `TaskConfiguration`
- **Optional**: `WorkerLibrary` for dynamic loading

Submit it to `SessionHandle`. This is **non-blocking** and returns a `TaskHandle` immediately.

---

## WorkerLibrary (Dynamic Loading)

`WorkerLibrary` describes a dynamically loaded worker library for task execution. It has three components:

| Field | Description | Example |
|---|---|---|
| `path` | Path to the artifact inside the zip | `"MyWorker.jar"` |
| `symbol` | Fully qualified class name | `"com.example.MyProcessor"` |
| `libraryHandle` | `BlobHandle` referencing the uploaded library zip | — |

---

## Typical Workflow

```
1. Create ArmoniKClient with config
2. Create SessionDefinition (with partitions, TaskConfiguration, listener)
3. Open SessionHandle
4. Define InputBlobDefinition and OutputBlobDefinition
5. Create and submit TaskDefinition
6. Wait for completion or react to BlobCompletionListener callbacks
7. Read output blobs
8. Close session and client
```

---

## Task Submission Flow (General)

1. The client creates a **session** to group related tasks.
2. Input data is uploaded as **blobs**.
3. The client defines one or more **task definitions**, each referencing:
- input blobs (data dependencies)
- expected output blobs
- optional task configuration
4. Tasks are submitted to the control plane.
5. Workers execute tasks, read inputs, and produce outputs.
6. The control plane signals task completion.
7. The client retrieves output blobs or submits new tasks to continue the workflow.
Loading
Loading