From b517528ce130b59b274a6006b2cc37e82f9cd09e Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Thu, 16 Apr 2026 14:55:35 -0600 Subject: [PATCH 1/3] Document replication settings, working directory, and latency metric --- reference/analytics/overview.md | 13 +++++++------ reference/components/javascript-environment.md | 4 ++++ reference/database/schema.md | 18 ++++++++++-------- 3 files changed, 21 insertions(+), 14 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 185c5df0..b3f4fb68 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -161,12 +161,13 @@ Harper automatically tracks the following metrics for all services. Applications ### Replication Metrics -| `metric` | `path` | `method` | `type` | Unit | Description | -| ---------------- | ------------- | ------------- | --------- | ----- | ----------------------------------- | -| `bytes-sent` | node.database | `replication` | `egress` | bytes | Bytes sent for replication | -| `bytes-sent` | node.database | `replication` | `blob` | bytes | Bytes sent for blob replication | -| `bytes-received` | node.database | `replication` | `ingress` | bytes | Bytes received for replication | -| `bytes-received` | node.database | `replication` | `blob` | bytes | Bytes received for blob replication | +| `metric` | `path` | `method` | `type` | Unit | Description | +| --------------------- | ------------------- | ------------- | --------- | ----- | ---------------------------------------------------------- | +| `bytes-sent` | node.database | `replication` | `egress` | bytes | Bytes sent for replication | +| `bytes-sent` | node.database | `replication` | `blob` | bytes | Bytes sent for blob replication | +| `bytes-received` | node.database | `replication` | `ingress` | bytes | Bytes received for replication | +| `bytes-received` | node.database | `replication` | `blob` | bytes | Bytes received for blob replication | +| `replication-latency` | node.database.table | | `ingest` | ms | Time difference from source commit timestamp to local time | ### Resource Usage Metrics diff --git a/reference/components/javascript-environment.md b/reference/components/javascript-environment.md index e93c566a..f5f0bb55 100644 --- a/reference/components/javascript-environment.md +++ b/reference/components/javascript-environment.md @@ -133,3 +133,7 @@ export class MyResource extends Resource { ### `getResponse()` Returns the outgoing `Response` object for the current request, or `undefined` if called outside a request context. Use this to set response headers or inspect the response mid-handler. Equivalent to `getContext().response`. + +### Current Working Directory + +Harper using a multi-threaded server architecture and uses the harper data root path as the current working directory. Components should not and cannot change the current working directory. diff --git a/reference/database/schema.md b/reference/database/schema.md index 08921f95..2ebb2264 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -73,12 +73,12 @@ type MyTable @table { Optional arguments: -| Argument | Type | Default | Description | -| ------------ | --------- | -------------- | ----------------------------------------------------------------------- | -| `table` | `String` | type name | Override the table name | -| `database` | `String` | `"data"` | Database to place the table in | -| `expiration` | `Int` | — | Auto-expire records after this many seconds (useful for caching tables) | -| `audit` | `Boolean` | config default | Enable audit log for this table | +| Argument | Type | Default | Description | +| ------------ | --------- | --------- | ----------------------------------------------------------------------- | +| `table` | `String` | type name | Override the table name | +| `database` | `String` | `"data"` | Database to place the table in | +| `expiration` | `Int` | — | Auto-expire records after this many seconds (useful for caching tables) | +| `replicate` | `Boolean` | true | Enable replication of this table | **Examples:** @@ -99,8 +99,8 @@ type Session @table(expiration: 3600) { userId: String } -# Enable audit log for this table explicitly -type AuditedRecord @table(audit: true) { +# Disable replication for this table explicitly +type LocalRecord @table(replicate: false) { id: Long @primaryKey value: String } @@ -114,6 +114,8 @@ type Event @table(database: "analytics", expiration: 86400) { **Database naming:** Since all tables default to the `data` database, when designing plugins or applications, consider using unique database names to avoid table naming collisions. +**Replication:** Replication is enabled by default for all tables. Note that if you disable replication on a table and re-enable it later, it will not catch-up on previous writes during when the replication was disabled. + ### `@export` Exposes the table as an externally accessible resource endpoint, available via REST, MQTT, and other interfaces. From 93fb954a73c54a41c57de0b8fb7b95323d0a9047 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 17 Apr 2026 13:46:59 -0600 Subject: [PATCH 2/3] Warn against process.chdir() --- reference/components/javascript-environment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/components/javascript-environment.md b/reference/components/javascript-environment.md index f5f0bb55..3cc8b317 100644 --- a/reference/components/javascript-environment.md +++ b/reference/components/javascript-environment.md @@ -136,4 +136,4 @@ Returns the outgoing `Response` object for the current request, or `undefined` i ### Current Working Directory -Harper using a multi-threaded server architecture and uses the harper data root path as the current working directory. Components should not and cannot change the current working directory. +Harper has a multi-threaded server architecture and uses the harper data root path as the current working directory. Components should not and cannot change the current working directory, and must not use `process.chdir()` or any package that does. From d073af0829b7857bf6d748f5c65861fdf48363ed Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 17 Apr 2026 15:42:31 -0600 Subject: [PATCH 3/3] Describe ExtendedIterable --- reference/resources/resource-api.md | 50 ++++++++++++++++++++++++++--- 1 file changed, 45 insertions(+), 5 deletions(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 5990d8d3..9a323ce3 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -25,7 +25,7 @@ Resource classes have static methods that directly map to RESTful methods or HTT Static methods are defined on a Resource class and called when requests are routed to the resource. This is the preferred way to interact with tables and resources from application code. You can override these methods to define custom behavior for these methods and for HTTP requests. -### `get(target: RequestTarget | Id, context?: Resource | Context): Promise | AsyncIterable` +### `get(target: RequestTarget | Id, context?: Resource | Context): Promise | ExtendedIterable` This can be called to retrieve a record by primary key. @@ -36,7 +36,7 @@ const product = await Product.get(34); The default `get` method returns a `RecordObject` — a frozen plain object with the record's properties plus `getUpdatedTime()` and `getExpiresAt()`. The record object is immutable because it represents the current state of the record in the database. -`get` is also called for HTTP GET requests and is always called with a `RequestTarget` as the `target` parameter. When the request targets a single record (e.g. `/Table/some-id`), the default `get` returns a single record object. When the request targets a collection (e.g. `/Table/?name=value`), the `target.isCollection` property is `true` and the default behavior calls `search()`, returning an `AsyncIterable`. +`get` is also called for HTTP GET requests and is always called with a `RequestTarget` as the `target` parameter. When the request targets a single record (e.g. `/Table/some-id`), the default `get` returns a single record object. When the request targets a collection (e.g. `/Table/?name=value`), the `target.isCollection` property is `true` and the default behavior calls `search()`, returning an `ExtendedIterable`. ```javascript class MyResource extends Resource { @@ -70,9 +70,9 @@ The `get()` method returns a `RecordObject` — a frozen plain object with all r --- -### `search(query: RequestTarget): AsyncIterable` +### `search(query: RequestTarget): ExtendedIterable` -`search` performs a query on the resource or table. This is called by `get()` on collection requests and can be overridden to define custom query behavior. The default implementation on tables queries by the `conditions`, `limit`, `offset`, `select`, and `sort` properties parsed from the URL. See [Query Object](#query-object) below for available query options. +`search` performs a query on the resource or table. This is called by `get()` on collection requests and can be overridden to define custom query behavior. The default implementation on tables queries by the `conditions`, `limit`, `offset`, `select`, and `sort` properties parsed from the URL. See [Query Object](#query-object) below for available query options. See the [ExtendedIterable](#extendediterable) below for how to interact with the query results. ### `put(target: RequestTarget | Id, data: Promise, context?: Resource | Context): Promise | Response` @@ -298,7 +298,7 @@ Publish a message to a record/topic. Subscribe to record changes or messages. -### `search(query: RequestTarget | Query, context?): AsyncIterable` +### `search(query: RequestTarget | Query, context?): ExtendedIterable` Query the table. See [Query Object](#query-object) below for available query options. @@ -736,6 +736,46 @@ The `get()` method returns a `RecordObject` — a frozen plain object with all r --- +## ExtendedIterable + +The `ExtendedIterable` extends and behaves like an `AsyncIterable`, but also includes a set of array-like methods: + +- `map`: Returns a new `ExtendedIterable` with the results of calling a provided function on every element in the calling `ExtendedIterable` (lazily evaluated as the iterable is consumed). +- `filter`: Returns a new `ExtendedIterable` with the elements that pass the test implemented by the provided function (lazily evaluated as the iterable is consumed). +- `flatMap`: Returns a new `ExtendedIterable` with the results of calling a provided function on every element in the calling `ExtendedIterable` and then flattening the result by one-level (lazily evaluated as the iterable is consumed). +- `concat`: Returns a new `ExtendedIterable` that contains the elements of the calling `ExtendedIterable` followed by the elements of the iterable passed as an argument (lazily evaluated as the iterable is consumed). +- `forEach`: Iterates the `ExtendedIterable`, calling the provided function once per element. This is executed eagerly/immediately. +- `slice`: Returns a new `ExtendedIterable` containing a subset of the elements of the calling `ExtendedIterable` (lazily evaluated as the iterable is consumed). +- `mapError`: Returns a new `ExtendedIterable` with that matches the calling `ExtendedIterable`, but maps any element evaluation that throws an error (lazily evaluated as the iterable is consumed) to a new value. + +These methods are intended to allow you to easily interact with the results of search queries, without having to convert the `ExtendedIterable` to an array. Generally, converting results to an array is discouraged because it can consume a excessive memory for large results, and undermines Harper's efficient iteration/streaming system. For example, you might write a `get` method like: + +```javascript +static async function get(target) { + const records = this.search(target); + // we can filter records here + const filteredRecords = records.map((record) => record.quantity > 100); + // we can map to new values + const mappedRecords = filteredRecords.map((record) => ({ ...record, extraProperty: 'value' })); + // we never converted this to an array, large results can efficiently to be streamed to the client + return mappedRecords; +} +``` + +If we do want to iterate the results within a function using a for-loop, you can use the `for await` syntax: + +```javascript +for await (const record of records) { + if (record.name === 'I found what I was looking for') { + return record; + } +} +``` + +(but again, using a for-loop to convert to an array is discouraged) + +See the [ExtendedIterable documentation](https://github.com/harperfast/extended-iterable) for more details. + ## Response Object Resource methods can return: