Add spec-based integration flow

.gitignore

@@ -94,4 +94,5 @@ src/pending_verification_prompts.json
 
 
 src/logs/*
-logs/*
+logs/*
+**/node_modules/

.python-version

@@ -0,0 +1 @@
+3.13

.vscode/settings.json

@@ -1,6 +1,6 @@
 {
-  "python.terminal.activateEnvironment": true,
-  "python.terminal.activateEnvInCurrentTerminal": true,
   "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
-  "python.analysis.extraPaths": ["${workspaceFolder}/src"]
+  "python.analysis.extraPaths": [
+    "${workspaceFolder}/src"
+  ]
 }

CONTEXT_LOG.md

@@ -0,0 +1,3 @@
+# Runtime Compatibility
+
+- The dependency lock currently resolves `pydantic-core==2.33.2`, which ships Python 3.13 wheels but falls back to a PyO3 source build on Python 3.14. PyO3 0.24.1 rejects 3.14, so the project needs to stay pinned to Python 3.13 until the dependency set is upgraded to a 3.14-compatible release line.

INTERVIEW.md

@@ -1,26 +0,0 @@
-# TODO 7: Multiline grouping approach (excluded from LLM body, protected from patches)
-
-> What exact on-disk syntax do you want for the multiline grouping section? Please provide a concrete before/after example (including where it sits relative to `---` front matter and the `# -- SCRATCHPAD` heading).
-
-after the front matter, before the scratchpad heading. it should be before any other content in the file after the front matter
-
-> How should the end of the multiline grouping section be detected (e.g., first blank line, next heading, a closing marker, end-of-file)? Can the grouping text itself contain blank lines?
-
-use explicit opening and closing marker syntax. if markdown frontmatter satisfies the criteria e.g. support for multiline content, use that. otherwise implement this in the way which makes the most sense / is the most aligned w/ good practice.
-
-> For legacy documents that still use a single-line `Grouping approach: ...` prefix, should the tool leave that line as-is, or migrate it to the new multiline format when writing the file? If migration is desired, should it happen only when `--grouping` is provided / user is prompted, or always?
-
-leave as is 
-
-> The current CLI prompt uses `input()` (single line). How do you want multiline grouping input to be entered (e.g., read until a lone `.` line, read until EOF, open $EDITOR, allow literal `\n` escapes, etc.)?
-
-yes figure out best/most simple but also useable way to support multiline input
-
-> Should the grouping section be preserved verbatim (whitespace/indentation), or normalized (trim lines, collapse spaces) before inserting into the prompt’s “Maintain the grouping approach: …” line?
-
-preserved verbatim 
-
-> Do you want the grouping section to be strictly immutable during patch application (i.e., patches only apply to the body after removing the grouping block), or should we also detect and error if a patch’s SEARCH text matches inside the grouping section?
-
-strictly immutable. it should be as if it didn't exist in the body at all. so if a search block matches it and nothing else, it results in an error > retry. there should not need to be any special handling logic for these cases. it simply isn't part of the document body for the purposes of search/replace or substitute blocks.
-

TODO.md

@@ -1,7 +1,13 @@
 - ask model to provide snippets from integrate text for each find/replace, to clarify what it intended it to integrate
     - on fail, only ask the model to provide just that single failed block instead of asking it to provide all blocks again. this is only possible once the model returns what integrated text each block relates 
 - only ask for start and end lines of search block instead of exact text. if there are multiple matches, ask model to provide a sufficient number of of lines at start or end to narrow down options to a single match. ensure that the search block which the verification prompt sees is not affected by this, by populating the SEARCH section of the block given in the verification prompt with the matching text from the file, instead of only including the start and end lines provided by the model
-- modify so that it uses tool calling + hierarchical markdown parsing to avoid needing to ever send the entire document to the model, and instead allow the model to find the relevant section(s) (could often be more than one section which should be modified even to integrate a single piece of info) to modify, and then once it has found the sections, it provides the search/replace diffs
-    - ensure that the model uses arbitrarily nested md headers, to make this approach scalable instead of only e.g. using one level of headings
-- move logs outside of src/
-- put group strat into front matter
+
+
+- move all files in this repo except integrate_notes.py to a new repo next to this one called zettel_inbox. still make sure to leave a copy of all non-code files in this repo, while also carrying them across.
+- add a new tool, in addition to find/replace, for creating a new file. this just takes a file name to create and initial text to add. add to the instructions an explanation that if the model wants to put some of the content from the chunk into a new file and link to it, because it couldn't find anywhere else satisfactory, it should just use the file tool to create the file, and then link to the file using a wikilink [[file_name]]. importantly it must always link to any new files it creates in at least one place (add validation rule for this). enforce that it must write some new text to any new file.
+
+- figure out how to make my notes work with butter context repo. Add symlinks automatically for all files under infofi into a dir in the butter repo. Check that this supports two way sync
+- add guidance re maximum size of file
+- add a copy and paste after function. also log all actions
+- prevent it from adding text to index gists. Only new files
+- use gemini flash

pyproject.toml

@@ -12,7 +12,7 @@ dependencies = [
     "requests",
     "openai>=1.109.1",
 ]
-requires-python = ">=3.13"
+requires-python = ">=3.13,<3.14"
 license = {text = "UNLICENCE"}
 readme = "README.md"
 

src/integrate_notes.py

@@ -201,7 +201,10 @@ def extract_grouping_section(body: str) -> tuple[GroupingSection | None, str]:
     while grouping_index < len(lines) and not lines[grouping_index].strip():
         grouping_index += 1
 
-    if grouping_index < len(lines) and lines[grouping_index].strip() == GROUPING_BLOCK_START:
+    if (
+        grouping_index < len(lines)
+        and lines[grouping_index].strip() == GROUPING_BLOCK_START
+    ):
         end_index = grouping_index + 1
         while end_index < len(lines) and lines[end_index].strip() != GROUPING_BLOCK_END:
             end_index += 1
@@ -242,7 +245,9 @@ def _format_grouping_block(grouping_text: str) -> str:
 
 
 def render_grouping_section(
-    grouping_text: str, existing_section: GroupingSection | None, preserve_existing: bool
+    grouping_text: str,
+    existing_section: GroupingSection | None,
+    preserve_existing: bool,
 ) -> str:
     if not grouping_text.strip():
         raise ValueError("Grouping approach cannot be empty.")
@@ -287,7 +292,7 @@ def prompt_for_grouping() -> str:
         f"{GROUPING_PREFIX} at the top of the document.\n"
         "Enter multiline text and finish with a single line containing only a '.'.\n"
         "Examples:\n"
-        "- Grouping approach: Group points according to what problem each idea/proposal/mechanism/concept addresses/are trying to solve, which you will need to figure out yourself based on context. Do not combine multiple goals/problems into one group. Keep goals/problems specific. Ensure groups are mutually exclusive and collectively exhaustive. Avoid overlap between group's goals/problems. sub-headings should be per-mechanism/per-solution i.e. according to which \"idea\"/solution each point relates to.\n"
+        '- Grouping approach: Group points according to what problem each idea/proposal/mechanism/concept addresses/are trying to solve, which you will need to figure out yourself based on context. Do not combine multiple goals/problems into one group. Keep goals/problems specific. Ensure groups are mutually exclusive and collectively exhaustive. Avoid overlap between group\'s goals/problems. sub-headings should be per-mechanism/per-solution i.e. according to which "idea"/solution each point relates to.\n'
         "- Group points according to what you think the most useful/interesting/relevant groupings are. Ensure similar, related and contradictory points are adjacent.\n"
         "Your input:\n"
     )
@@ -1391,7 +1396,9 @@ def integrate_notes(
     source_body, source_scratchpad = split_document_sections(source_content)
     grouping_section, working_body = extract_grouping_section(source_body)
 
-    resolved_grouping = grouping or (grouping_section.text if grouping_section else None)
+    resolved_grouping = grouping or (
+        grouping_section.text if grouping_section else None
+    )
     if not resolved_grouping:
         resolved_grouping = prompt_for_grouping()
         logger.info("Recorded new grouping approach from user input.")