Document replication settings, working directory, and latency metric

reference/analytics/overview.md

@@ -161,12 +161,13 @@ Harper automatically tracks the following metrics for all services. Applications
 
 ### Replication Metrics
 
-| `metric`         | `path`        | `method`      | `type`    | Unit  | Description                         |
-| ---------------- | ------------- | ------------- | --------- | ----- | ----------------------------------- |
-| `bytes-sent`     | node.database | `replication` | `egress`  | bytes | Bytes sent for replication          |
-| `bytes-sent`     | node.database | `replication` | `blob`    | bytes | Bytes sent for blob replication     |
-| `bytes-received` | node.database | `replication` | `ingress` | bytes | Bytes received for replication      |
-| `bytes-received` | node.database | `replication` | `blob`    | bytes | Bytes received for blob replication |
+| `metric`              | `path`              | `method`      | `type`    | Unit  | Description                                                |
+| --------------------- | ------------------- | ------------- | --------- | ----- | ---------------------------------------------------------- |
+| `bytes-sent`          | node.database       | `replication` | `egress`  | bytes | Bytes sent for replication                                 |
+| `bytes-sent`          | node.database       | `replication` | `blob`    | bytes | Bytes sent for blob replication                            |
+| `bytes-received`      | node.database       | `replication` | `ingress` | bytes | Bytes received for replication                             |
+| `bytes-received`      | node.database       | `replication` | `blob`    | bytes | Bytes received for blob replication                        |
+| `replication-latency` | node.database.table |               | `ingest`  | ms    | Time difference from source commit timestamp to local time |
 
 ### Resource Usage Metrics
 

reference/components/javascript-environment.md

@@ -133,3 +133,7 @@ export class MyResource extends Resource {
 ### `getResponse()`
 
 Returns the outgoing `Response` object for the current request, or `undefined` if called outside a request context. Use this to set response headers or inspect the response mid-handler. Equivalent to `getContext().response`.
+
+### Current Working Directory
+
+Harper has a multi-threaded server architecture and uses the harper data root path as the current working directory. Components should not and cannot change the current working directory, and must not use `process.chdir()` or any package that does.

reference/database/schema.md

@@ -80,7 +80,7 @@ Optional arguments:
 | `expiration`   | `Int`     | —                             | Seconds until a record goes stale (useful for caching tables)               |
 | `eviction`     | `Int`     | `0`                           | Additional seconds after `expiration` before a record is physically removed |
 | `scanInterval` | `Int`     | `(expiration + eviction) / 4` | Seconds between eviction scans                                              |
-| `audit`        | `Boolean` | config default                | Enable audit log for this table                                             |
+| `replicate`    | `Boolean` | true                          | Enable replication of this table                                            |
 
 **`expiration`, `eviction`, and `scanInterval`**
 
@@ -145,8 +145,8 @@ type Session @table(expiration: 3600) {
 	userId: String
 }
 
-# Enable audit log for this table explicitly
-type AuditedRecord @table(audit: true) {
+# Disable replication for this table explicitly
+type LocalRecord @table(replicate: false) {
 	id: Long @primaryKey
 	value: String
 }
@@ -160,6 +160,8 @@ type Event @table(database: "analytics", expiration: 86400) {
 
 **Database naming:** Since all tables default to the `data` database, when designing plugins or applications, consider using unique database names to avoid table naming collisions.
 
+**Replication:** Replication is enabled by default for all tables. Note that if you disable replication on a table and re-enable it later, it will not catch-up on previous writes during when the replication was disabled.
+
 ### `@export`
 
 Exposes the table as an externally accessible resource endpoint, available via REST, MQTT, and other interfaces.

reference/resources/resource-api.md

@@ -25,7 +25,7 @@ Resource classes have static methods that directly map to RESTful methods or HTT
 
 Static methods are defined on a Resource class and called when requests are routed to the resource. This is the preferred way to interact with tables and resources from application code. You can override these methods to define custom behavior for these methods and for HTTP requests.
 
-### `get(target: RequestTarget | Id, context?: Resource | Context): Promise<object> | AsyncIterable`
+### `get(target: RequestTarget | Id, context?: Resource | Context): Promise<object> | ExtendedIterable`
 
 This can be called to retrieve a record by primary key.
 
@@ -36,7 +36,7 @@ const product = await Product.get(34);
 
 The default `get` method returns a `RecordObject` — a frozen plain object with the record's properties plus `getUpdatedTime()` and `getExpiresAt()`. The record object is immutable because it represents the current state of the record in the database.
 
-`get` is also called for HTTP GET requests and is always called with a `RequestTarget` as the `target` parameter. When the request targets a single record (e.g. `/Table/some-id`), the default `get` returns a single record object. When the request targets a collection (e.g. `/Table/?name=value`), the `target.isCollection` property is `true` and the default behavior calls `search()`, returning an `AsyncIterable`.
+`get` is also called for HTTP GET requests and is always called with a `RequestTarget` as the `target` parameter. When the request targets a single record (e.g. `/Table/some-id`), the default `get` returns a single record object. When the request targets a collection (e.g. `/Table/?name=value`), the `target.isCollection` property is `true` and the default behavior calls `search()`, returning an `ExtendedIterable`.
 
 ```javascript
 class MyResource extends Resource {
@@ -70,9 +70,9 @@ The `get()` method returns a `RecordObject` — a frozen plain object with all r
 
 ---
 
-### `search(query: RequestTarget): AsyncIterable`
+### `search(query: RequestTarget): ExtendedIterable`
 
-`search` performs a query on the resource or table. This is called by `get()` on collection requests and can be overridden to define custom query behavior. The default implementation on tables queries by the `conditions`, `limit`, `offset`, `select`, and `sort` properties parsed from the URL. See [Query Object](#query-object) below for available query options.
+`search` performs a query on the resource or table. This is called by `get()` on collection requests and can be overridden to define custom query behavior. The default implementation on tables queries by the `conditions`, `limit`, `offset`, `select`, and `sort` properties parsed from the URL. See [Query Object](#query-object) below for available query options. See the [ExtendedIterable](#extendediterable) below for how to interact with the query results.
 
 ### `put(target: RequestTarget | Id, data: Promise<object>, context?: Resource | Context): Promise<void> | Response`
 
@@ -298,7 +298,7 @@ Publish a message to a record/topic.
 
 Subscribe to record changes or messages.
 
-### `search(query: RequestTarget | Query, context?): AsyncIterable`
+### `search(query: RequestTarget | Query, context?): ExtendedIterable`
 
 Query the table. See [Query Object](#query-object) below for available query options.
 
@@ -736,6 +736,46 @@ The `get()` method returns a `RecordObject` — a frozen plain object with all r
 
 ---
 
+## ExtendedIterable
+
+The `ExtendedIterable` extends and behaves like an `AsyncIterable`, but also includes a set of array-like methods:
+
+- `map`: Returns a new `ExtendedIterable` with the results of calling a provided function on every element in the calling `ExtendedIterable` (lazily evaluated as the iterable is consumed).
+- `filter`: Returns a new `ExtendedIterable` with the elements that pass the test implemented by the provided function (lazily evaluated as the iterable is consumed).
+- `flatMap`: Returns a new `ExtendedIterable` with the results of calling a provided function on every element in the calling `ExtendedIterable` and then flattening the result by one-level (lazily evaluated as the iterable is consumed).
+- `concat`: Returns a new `ExtendedIterable` that contains the elements of the calling `ExtendedIterable` followed by the elements of the iterable passed as an argument (lazily evaluated as the iterable is consumed).
+- `forEach`: Iterates the `ExtendedIterable`, calling the provided function once per element. This is executed eagerly/immediately.
+- `slice`: Returns a new `ExtendedIterable` containing a subset of the elements of the calling `ExtendedIterable` (lazily evaluated as the iterable is consumed).
+- `mapError`: Returns a new `ExtendedIterable` with that matches the calling `ExtendedIterable`, but maps any element evaluation that throws an error (lazily evaluated as the iterable is consumed) to a new value.
+
+These methods are intended to allow you to easily interact with the results of search queries, without having to convert the `ExtendedIterable` to an array. Generally, converting results to an array is discouraged because it can consume a excessive memory for large results, and undermines Harper's efficient iteration/streaming system. For example, you might write a `get` method like:
+
+```javascript
+static async function get(target) {
+	const records = this.search(target);
+	// we can filter records here
+	const filteredRecords = records.map((record) => record.quantity > 100);
+	// we can map to new values
+	const mappedRecords = filteredRecords.map((record) => ({ ...record, extraProperty: 'value' }));
+	// we never converted this to an array, large results can efficiently to be streamed to the client
+	return mappedRecords;
+}
+```
+
+If we do want to iterate the results within a function using a for-loop, you can use the `for await` syntax:
+
+```javascript
+for await (const record of records) {
+	if (record.name === 'I found what I was looking for') {
+		return record;
+	}
+}
+```
+
+(but again, using a for-loop to convert to an array is discouraged)
+
+See the [ExtendedIterable documentation](https://github.com/harperfast/extended-iterable) for more details.
+
 ## Response Object
 
 Resource methods can return: