[PATCH] allow a tokeniser to use TStrings if it wants to

[andrewd207]

I started writing a sort of code aware parser to see if I could but it uses the text of the edit directly.
Yours is much faster 😄 But anyway, maybe it doesn't hurt to be able to use the Lines directly. By default it calls Tokenise(Lines.Text) so there's no fundamental change.

As a postscript, The one I wrote is keeping track of the state and reporting errors it finds. I'll share it in the coming days.

diff --git a/ide/src/main/pascal/ide.highlight.renderer.pas b/ide/src/main/pascal/ide.highlight.renderer.pas
index d4ece69a..c0da9912 100644
--- a/ide/src/main/pascal/ide.highlight.renderer.pas
+++ b/ide/src/main/pascal/ide.highlight.renderer.pas
@@ -154,7 +154,7 @@ procedure THighlighterCache.EnsurePascalTokenised(AEditor: TObject; ALines: TStr
   begin
     FPascalEditor := AEditor;
     if Assigned(ALines) then
-      FPascalHighlighter.Tokenise(ALines.Text);
+      FPascalHighlighter.Tokenise(ALines);
   end;
 end;
 
diff --git a/ide/src/main/pascal/ide.highlighter.pas b/ide/src/main/pascal/ide.highlighter.pas
index 446ca200..a36674de 100644
--- a/ide/src/main/pascal/ide.highlighter.pas
+++ b/ide/src/main/pascal/ide.highlighter.pas
@@ -83,10 +83,12 @@   TEditorHighlighter = class(TObject)
     procedure AddToken(ALine, AColumn, ALength: Integer; ACategory: THighlightCategory);
     procedure EnsureLineCount(ACount: Integer);
     procedure DoTokenise(const AText: string); virtual; abstract;
+    procedure DoTokenise(const ALines: TStrings); virtual;
   public
     constructor Create; virtual;
     destructor Destroy; override;
     procedure Tokenise(const AText: string);
+    procedure Tokenise(const ALines: TStrings);
     function LineCount: Integer;
     function GetLineTokens(ALine: Integer): THighlightTokenArray;
     function GetLineTokenCount(ALine: Integer): Integer;
@@ -157,6 +159,11 @@ procedure TEditorHighlighter.EnsureLineCount(ACount: Integer);
   end;
 end;
 
+procedure TEditorHighlighter.DoTokenise(const ALines: TStrings);
+begin
+  DoTokenise(ALines.Text);
+end;
+
 procedure TEditorHighlighter.AddToken(ALine, AColumn, ALength: Integer;
   ACategory: THighlightCategory);
 var
@@ -186,6 +193,13 @@ procedure TEditorHighlighter.Tokenise(const AText: string);
     DoTokenise(AText);
 end;
 
+procedure TEditorHighlighter.Tokenise(const ALines: TStrings);
+begin
+  ClearLines;
+  if ALines.Count > 0 then
+    DoTokenise(ALines);
+end;
+
 function TEditorHighlighter.LineCount: Integer;
 begin
   Result := FLines.Count;

[graemeg]

Thanks for the patch and glad I did something right regarding the
tokeniser speed! :)

I think the tokeniser is best kept as a pure text-stream tool — it
doesn't really have a concept of lines, so a TStrings interface would
be a bit misleading about what it does under the hood (consumes a
single string and advances through it character-by-character). The
other thing is that the .Text allocation still happens inside the
default DoTokenise(TStrings) override, so there's no real cost saving
at the call site either.

Your stateful error-tracking parser sounds really interesting though.

[andrewd207]

Thanks for the patch and glad I did something right regarding the tokeniser speed! :)

I think the tokeniser is best kept as a pure text-stream tool — it doesn't really have a concept of lines, so a TStrings interface would be a bit misleading about what it does

It's still just using the TStrings as an array to get each line. It doesn't seem that confusing. And each token is also given as a line/column position.

The only reason it's calling lines.text in the default is because the existing parsers expect it in that format so it remains compatible/the same.

Also as a bonus, I was connecting to the observer notifications of the TStrings to limit parsing on unaffected text. Although that's something in progress. I.e. when text on a single line is changed it just call TStrings.Put.
So with TStrings I know exactly where the line is that's modified and it doesn't have to be parsed to find it. I just started at the closest previous function start before the modified line.

Your stateful error-tracking parser sounds really interesting though.

Essentially as it gives each token it also pases the path to arrive there. Unit.myunit.interface.type.TMType.private.function.MyFunction.Params.myparam.string etc

That's the part I thought would be most useful too. It's meant to be pretty lax but provide errors of a couple severities.

Soooo, if its not too egregious maybe having an option for TStrings is acceptable. If one parser only implements as a single String it can use that. Or as TStrings it can use it directly. Would you reconsider ? 🤞