diff --git a/config.yaml b/config.yaml index 33e0fc7..da79921 100644 --- a/config.yaml +++ b/config.yaml @@ -16,3 +16,7 @@ jsResource: # Serve the static files from the web directory as a web application static: files: 'web/*' + +# Loads data into user tables from YAML or JSON files +dataLoader: + files: data/*.{json,yaml,yml} diff --git a/data/README.md b/data/README.md new file mode 100644 index 0000000..fc1e048 --- /dev/null +++ b/data/README.md @@ -0,0 +1,67 @@ +# HarperDB Data Loader + +This directory contains YAML or JSON files that are automatically loaded into your HarperDB database tables when your application starts. + +## How It Works + +1. Place data files in this directory with `.yaml`, `.yml`, or `.json` extensions +2. Files are processed when HarperDB starts +3. Records are inserted/updated based on file modification time: + - New records are always added + - Existing records are only updated if the file's modification time is newer than the record's stored timestamp + - Records with timestamps newer than the file's modification time are preserved unchanged + +## File Format + +Data files must contain `table` and `records` fields. The `database` field is optional. + +### JSON Example + +```jsonc +{ + "database": "dev", // Optional - uses default database if omitted + "table": "Product", // Required - name of the table + "records": [ // Required - array of records + { + "id": "1", // Primary key field + "name": "Laptop", + "price": 999.99 + } + ] +} +``` + +### YAML Example + +```yaml +database: dev # Optional - uses default database if omitted +table: products # Required - name of the table +records: # Required - array of records + - id: 1 # Primary key field + name: "Laptop" + price: 999.99 +``` + +## Key Features + +- **Automatic Table Creation**: Tables are created if they don't exist +- **Schema Inference**: + - The system will automatically infer a schema from the first record when creating a new table + - The `id` field is automatically detected as the primary key + - It's preferred to define the schema explicitly in your application, but this provides a quick start +- **File Modification Time**: + - The data loader uses the file's modification time (`mtime`) to determine if records should be updated + - "Touching" a file (updating its modification time) will force a reload of its data +- **Multi-file Support**: You can have multiple data files for different tables +- **Complex Data Types**: Supports nested objects, arrays, and various data types +- **One Table Per File**: Each file should define one table + +## Tips for Managing Data + +- To force a reload of data, simply update the file's modification time: + ```bash + # Update the file's timestamp using touch + touch data/products.json + ``` +- If you need to restore to previous data, you can replace the file and update its timestamp +- The system automatically handles the comparison between file modification time and record timestamps \ No newline at end of file diff --git a/schema.graphql b/schema.graphql index 5ad5d48..29cc77f 100644 --- a/schema.graphql +++ b/schema.graphql @@ -1,5 +1,8 @@ -## Here we can define any tables in our database. This example shows how we define a type as a table using -## the type name as the table name and specifying it is an "export" available in the REST and other external protocols. +## This schema defines the data model for our application. +## Each type with the @table directive becomes a table in the database. +## The @export directive makes it available in the REST API and other external protocols. + +## Example table for reference type TableName @table @export { id: ID @primaryKey # Here we define primary key (must be one) name: String # we can define any other attributes here