Skip to content

ghcp(crawl): ex1 repo orientation + low-risk module #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RitaChen609 wants to merge 1 commit into
base: learn/crawl/ex0-bootstrap
Choose a base branch
from
Open

ghcp(crawl): ex1 repo orientation + low-risk module #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 83 additions & 12 deletions ai-track-docs/SYSTEM-OVERVIEW.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,89 @@

## Purpose

The **MarkLogic Java Client API** (`com.marklogic:marklogic-client-api`) is a Java library that exposes MarkLogic Server's REST API as a type-safe, fluent Java interface. It supports reading, writing, deleting, and querying JSON, XML, binary, and text documents, as well as ACID multi-statement transactions, semantic (SPARQL/RDF), Full-text search, alerting, Data Services, and Row Manager (Optic API).
The **MarkLogic Java Client API** (`com.marklogic:marklogic-client-api`) is a Java library that exposes MarkLogic Server's REST API as a type-safe, fluent Java interface. It supports reading, writing, deleting, and querying JSON, XML, binary, and text documents, as well as ACID multi-statement transactions, semantic (SPARQL/RDF), full-text search, alerting, Data Services, and Row Manager (Optic API).

## Repository Root

`marklogic-client-api-parent` (Gradle multi-project).

## Modules

| Module | Description |
| -------------------------------------- | ---------------------------------------------------------------------------- |
| `marklogic-client-api` | Core library — all production source code |
| `marklogic-client-api-functionaltests` | Functional / integration tests requiring a live MarkLogic instance |
| `ml-development-tools` | Kotlin-based developer tooling (code generation helpers) |
| `test-app` | MarkLogic application deployed to the test server (modules, schemas, config) |
| `examples` | Standalone usage examples |
| Module | Folder | Description |
| -------------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `marklogic-client-api` | `marklogic-client-api/` | Core library — all production source code (~616 Java files) |
| `marklogic-client-api-functionaltests` | `marklogic-client-api-functionaltests/` | Functional / integration tests — all require a live MarkLogic instance (147 files) |
| `ml-development-tools` | `ml-development-tools/` | Kotlin-based Gradle plugin + CLI tools for Data Services code generation (8 Kotlin source files) |
| `test-app` | `test-app/` | MarkLogic application deployed to the test server (modules, schemas, ML config) |
| `examples` | `examples/` | Standalone usage examples (63 Java files, no automated tests) |

## Languages

| Language | Location | Notes |
| ------------------ | ----------------------------------------------------------------------------------------- | ----------------------------------------------- |
| **Java** | `marklogic-client-api/src/`, `examples/src/`, `marklogic-client-api-functionaltests/src/` | Primary language (~997 source + test files) |
| **Kotlin** | `ml-development-tools/src/main/kotlin/` | Code generator and Gradle plugin only |
| **Server-Side JS** | `marklogic-client-api/src/test/resources/`, `ml-development-tools/src/test/` | Test transforms and module stubs (.sjs/.mjs) |
| **XQuery** | `marklogic-client-api/src/test/resources/` | Eval tests, rule transforms |
| **XML / JSON** | Throughout `src/test/resources/` and `test-app/src/` | Test fixtures, ML config, endpoint declarations |

## Public API Entry Points

All live under `marklogic-client-api/src/main/java/com/marklogic/client/`.

| Class | Path | Role |
| ----------------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `DatabaseClientBuilder` | `DatabaseClientBuilder.java` | **Preferred modern entry point** (since 6.1.0). Fluent builder: `.withHost()`, `.withPort()`, `.withBasicAuth()`, `.withConnectionString()`, `.build()`. |
| `DatabaseClientFactory` | `DatabaseClientFactory.java` | Older static factory; also hosts nested auth-context types (`BasicAuthContext`, `DigestAuthContext`, `CloudAuthContext`, etc.) and the Spring-friendly `Bean` class. |
| `DatabaseClient` | `DatabaseClient.java` | Central interface returned by both factory and builder. Gateway to all managers via `.newJSONDocumentManager()`, `.newQueryManager()`, `.newRowManager()`, `.newDataMovementManager()`, `.openTransaction()`, etc. |
| `QueryManager` / `StructuredQueryBuilder` | `query/` (33 classes) | Full-text and structured search DSL, obtained via `client.newQueryManager()`. |
| `DataMovementManager` | `datamovement/DataMovementManager.java` | Bulk/streaming operations. Creates `QueryBatcher` (bulk read/export) and `WriteBatcher` (bulk write). ~45 interfaces/classes in `datamovement/`. |

## Test Approach

- **Framework**: JUnit 5 throughout `marklogic-client-api` and `marklogic-client-api-functionaltests`; JUnit 4 in `ml-development-tools` (intentionally kept — noted in `ml-development-tools/build.gradle`).
- **Mocking**: Mockito present but minimal (2 files: `SPARQLQueryManagerImplTest.java`, `SSLTest.java`). OkHttp `MockWebServer` used for HTTP-level tests. The dominant pattern is a real MarkLogic connection via `Common.newClient()`.
- **Server dependency split** (core module, 214 test files): roughly half require a live MarkLogic instance; half run standalone.
- **Functional tests** (`marklogic-client-api-functionaltests/`): 100% server-dependent. MarkLogic is started via `docker-compose.yaml`.

> **Assumption**: The ~1:1 standalone/server split was estimated by scanning for `Common.connect`/`Common.newClient` imports. Verify with:
>
> ```powershell
> $t="marklogic-client-api\src\test\java"
> (Get-ChildItem -Path $t -Recurse -Filter "*.java" |
> Select-String "Common\.connect|Common\.newClient" |
> Group-Object Path | Measure-Object).Count
> ```

## Recommended Low-Risk Starting Point

**`MarkLogicVersion`** — a pure string-parsing utility with full standalone test coverage.

| File | Path |
| ------ | ---------------------------------------------------------------------------------------- |
| Source | `marklogic-client-api/src/test/java/com/marklogic/client/test/MarkLogicVersion.java` |
| Test | `marklogic-client-api/src/test/java/com/marklogic/client/test/MarkLogicVersionTest.java` |

Why it's the safest first touch:

- **No server, no Docker** — runs in seconds with no infrastructure.
- **Blast radius is zero** — parsing utility; nothing in the auth or network path depends on it.
- **Fast feedback loop** — 8 parameterised JUnit 5 tests cover the full version-string format.
- **Tiny surface area** — both files together are under 100 lines; ideal for getting comfortable with the project's code style and test conventions.

Run it in isolation:

```bash
./gradlew :marklogic-client-api:test --tests "com.marklogic.client.test.MarkLogicVersionTest"
```

Other low-risk standalone targets (no server needed):

| Class | Source path | Test path |
| ------------------------------ | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
| `PlanRowColTypesImpl` | `marklogic-client-api/src/main/java/com/marklogic/client/impl/PlanRowColTypesImpl.java` | `marklogic-client-api/src/test/java/com/marklogic/client/impl/PlanRowColTypesImplTest.java` |
| `TransformDefImpl` | `marklogic-client-api/src/main/java/com/marklogic/client/impl/TransformDefImpl.java` | `marklogic-client-api/src/test/java/com/marklogic/client/impl/TransformDefImplTest.java` |
| `DatabaseClientPropertySource` | `marklogic-client-api/src/main/java/com/marklogic/client/impl/DatabaseClientPropertySource.java` | `marklogic-client-api/src/test/java/com/marklogic/client/impl/DatabaseClientPropertySourceTest.java` |

## Runtime Requirements

Expand All @@ -26,16 +94,19 @@ The **MarkLogic Java Client API** (`com.marklogic:marklogic-client-api`) is a Ja
## Technology Stack

- Build: **Gradle** (wrapper at `./gradlew`)
- Test framework: **JUnit 5** (unit) + MarkLogic functional test harness
- Test framework: **JUnit 5** (core) / **JUnit 4** (`ml-development-tools`)
- CI: **Jenkins** (`Jenkinsfile`) — Docker-based MarkLogic, parallel Java 17/21 builds
- Primary language: **Java**; developer tooling in **Kotlin**

## Key External Dependencies

- OkHttp (HTTP client transport)
- Jackson (JSON serialization)
- OkHttp 3 (HTTP client transport)
- Jackson Databind + Jackson CSV (JSON serialization)
- SLF4J / Logback (logging)
- JAXB 4 / Glassfish runtime (XML binding)
- `okhttp-digest` (Digest auth)
- Optional: JDOM2, dom4j, Gson

## Relationship to MarkLogic Server

All network communication travels over the **MarkLogic REST Management and Client APIs** (typically port 8000/8002). The library never connects directly to MarkLogic's internal ports; authentication is via HTTP Digest or certificate.
All network communication travels over the **MarkLogic REST Client and Management APIs** (typically port 8000/8002). The library never connects to MarkLogic's internal ports directly; authentication is via HTTP Basic, Digest, certificate, Kerberos, SAML, or OAuth/Cloud token.