feed.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
<title type="text">Expressions of Change</title>
<generator uri="https://github.com/mojombo/jekyll">Jekyll</generator>
<link rel="self" type="application/atom+xml" href="http://www.expressionsofchange.org/feed.xml" />
<link rel="alternate" type="text/html" href="http://www.expressionsofchange.org" />
<updated>2025-01-20T22:33:08+01:00</updated>
<id>http://www.expressionsofchange.org/</id>
<author>
  <name>Klaas van Schelven</name>
  <uri>http://www.expressionsofchange.org/</uri>
  <email>klaas@vanschelven.com</email>
</author>


<entry>
  <title type="html"><![CDATA[Homoiconicity revisited]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/homoiconicity-revisited/" />
  <id>http://www.expressionsofchange.org/homoiconicity-revisited</id>
  <published>2020-06-02T00:00:00+02:00</published>
  <updated>2020-06-02T00:00:00+02:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;In an &lt;a href=&quot;/dont-say-homoiconic/&quot;&gt;earlier article&lt;/a&gt;, I concluded that you probably
shouldn’t use the word “homoiconic”: starting with the original definition of
the word, I noted that this this definition is problematic for a number of
reasons, and that the best we can say is that there is a degree to which a
language is homoiconic: languages that have a smaller conceptual distance
between their program text and machine operation are more homoiconic and
vice versa.&lt;/p&gt;

&lt;p&gt;That article also explored some of the plethora of competing, often mutually
exclusive definitions in active use – a fact that should come as a warning
anyone who’s intention is to clearly communicate their ideas.&lt;/p&gt;

&lt;p&gt;It is therefore somewhat disappointing (though not unsurprising) that when the
&lt;a href=&quot;https://news.ycombinator.com/item?id=20657798&quot;&gt;article was discussed on Hacker
News&lt;/a&gt;, the top voted comment
started like this:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;I’m going to keep using the word ‘homoiconic’ because it is a
useful term.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In style, that commenter neither explained why the term would be
useful, nor provided a definition of their own.&lt;/p&gt;

&lt;p&gt;Given that people will continue to use the word, what are we to understand if
they do? Let’s revisit the topic, but start at the other end: instead of
taking the official definition as a starting point, let’s say a word means what
the people who use it mean by it. Even if that meaning has little to do with
the original definition, and even if they are not willing to put forward a
definition of their own.&lt;/p&gt;

&lt;p&gt;If we don’t want to start with the official definition, we’re faced with a bit
of a challenge though: what are we to take as a starting point for the
discussion? To get it around it, we’ll simply start with a definition of our
own, and then match this back to typical examples and counter-examples of
homoiconicity to see how it fits.&lt;/p&gt;

&lt;h3 id=&quot;homoiconic-a-working-definition&quot;&gt;Homoiconic, a working definition&lt;/h3&gt;

&lt;p&gt;Homoiconic usually refers some combination of the following (weights assigned
to the bullets below will depend on the speaker):&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;
    &lt;p&gt;Strong language support for simple, composable (i.e. tree-like) data
structures, preferably using a minimal syntax (i.e. literals).&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;The language semantics are directly defined in terms of such data structures
and the programs are formed using instances of these structures.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;The structure of such data-structures is explicitly reflected in their
visual representation on-screen; it is immediately apparant &lt;em&gt;for humans&lt;/em&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;High similarlity between representations of the program in (some of):&lt;/p&gt;
    &lt;ul&gt;
      &lt;li&gt;the head of the programmer&lt;/li&gt;
      &lt;li&gt;the visual representation on-screen&lt;/li&gt;
      &lt;li&gt;the formal semantics of the language&lt;/li&gt;
      &lt;li&gt;the implementation of the (virtual) machine&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The benefit of homoiconicity is then: first, the structured manipulation of
programs, possibly by other programs, becomes trivial and natural. Second,
there is little mental effort spent in mapping the program text to the program
structure, and in imagining how the machine operates on said structure.&lt;/p&gt;

&lt;h3 id=&quot;examples-and-counter-examples&quot;&gt;Examples and counter-examples&lt;/h3&gt;

&lt;p&gt;Lisps tick all, or most of, of the boxes:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;S-expressions take the role of the simple, composable data structures.&lt;/li&gt;
  &lt;li&gt;Lisps’ semantics are directly defined in terms of s-expressions and
lisp programs are formed using s-expressions.&lt;/li&gt;
  &lt;li&gt;S-expressions are trees, and the nesting of items is immediately apparent:
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(&lt;/code&gt; denotes the start of a child, and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;)&lt;/code&gt; its end. Children are nested
inside their parents.&lt;/li&gt;
  &lt;li&gt;Lisp programmers think in s-expressions, which are always on-screen, and
form the basis of the semantics of the language (always) and the
implementation of the VM (if it is an interpreter).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Other languages tick only some of the boxes, or none of them.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;JavaScript has strong language support for composable, tree-like, structures
(bullet 1) and the structure is immediately apparent by looking at the pairs
of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;{&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;}&lt;/code&gt; brackets (bullet 3). However, the language semantics are not
defined in such terms (bullet 2) and a JavaScript program is not provided as
a JavaScript object (or JSON object).&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;For machine code, little mapping between the representations for the human
and the machine is required, for the simple reason that no special representation
for humans exist (bullet 4). However, machine code fails on bullets 1 to 3.
I’d argue that few people that praise the benefits of homoiconicity actually
think of machine code while doing so, though they might admit machine code
fits some definition of homoiconicity when pressed.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;XSLT is defined in XML, i.e. in composable, tree-like structures, so it gets
a pass on bullets 1, 2 &amp;amp; 3 (with the exception of “simple syntax”). This is
another one for “homoiconic, but not a poster child”.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;TRAC probably fails on bullets 1, 2 &amp;amp; 3, even though it was the language in
the context of which the term homoiconic was coined. This is simply a
reflection of the fact that the term has evolved, and that modern-day usage
is different from the original meaning.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Java has no strong language support for literal representation of simple
composable data in the language (bullet 1) and also fails to meet the
other criteria.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;what-lispers-dont-tell-you&quot;&gt;What Lispers don’t tell you&lt;/h3&gt;

&lt;p&gt;So is the above definition new in any way? I think it is, in the sense that is
spells out some things that might be so obvious to Lispers that they forget to
include them in their definitions. (“What’s water?” says the fish)&lt;/p&gt;

&lt;p&gt;Take for example the definition on &lt;a href=&quot;https://wiki.c2.com/?HomoiconicLanguages&quot;&gt;Wards Wiki&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;In a homoiconic language, the primary representation of programs is also a
data structure in a primitive type of the language itself.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A &lt;a href=&quot;https://wiki.c2.com/?HomoiconicExampleInJava&quot;&gt;common objection against this definition being
meaningful&lt;/a&gt; is that this is or
can be the case for most languages: e.g. in Java the primary representation of
programs is as text (strings), and strings are also a primitive type of the
language itself. Such an objection (understandably) glosses over the meaning
implied by “a datastructure in”.&lt;/p&gt;

&lt;p&gt;For non-lispers, that phrase might not evoke much, or may perhaps suggest an
implementation detail (since “data structure” is often used to refer to the
implementation of data types, i.e. a string might be implemented as an array,
in which case the array is the underlying datastructure).&lt;/p&gt;

&lt;p&gt;For Lispers, this part is essential though, because for them it is naturally a
reference to s-expressions. And thus for Lispers it evokes 2 parts of the
definition that I made explicit in the above under bullets 1 and 2, namely:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;the compositional nature of the data type under consideration&lt;/li&gt;
  &lt;li&gt;the fact that data and programs can be composed in the same way&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;As a second example, consider the opening line of the current definition on
&lt;a href=&quot;https://en.wikipedia.org/wiki/Homoiconicity&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;A language is homoiconic if a program written in it can be manipulated as
data using the language&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Here, again, the objection can be raised that this is true for all languages.
That is, all languages allow for programs written in them to be manipulated by
them “as data” (i.e. as strings). After all, strings are data.&lt;/p&gt;

&lt;p&gt;Well… not for Lispers, to whom “data” will likely evoke something more
structured, such as s-expressions (again). In other words, the key property
here is that the program &lt;em&gt;is&lt;/em&gt; a piece of &lt;em&gt;hierarchically structured&lt;/em&gt; data
(bullets 1 &amp;amp; 2 in the above).&lt;/p&gt;

&lt;p&gt;Finally, let’s consider how Wikipedia continues:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;and thus the program’s internal representation can be inferred just by
reading the program itself.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Here, the typical counterpoint would be that one can either never be true
(because the inner workings of the machine are hidden from us) or is always
true (if we assume that we are being provided the specification).&lt;/p&gt;

&lt;p&gt;Fitting this sentence back to bullets 3 &amp;amp; 4 makes more sense though: the key
point is not that the inner representation &lt;em&gt;can be inferred&lt;/em&gt;, but rather that
a “good enough model” is easy enough to imagine while looking at the program
text.&lt;/p&gt;

&lt;h3 id=&quot;homo-reinterpreted&quot;&gt;“Homo” reinterpreted&lt;/h3&gt;

&lt;p&gt;If we indeed accept the working definition in the above as what most people
who speak about homoiconicity actually mean, it seems that an interesting
shift has occurred. Take one more look at the original definition:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Because TRAC procedures and text have the same representation inside and
outside the processor, the term homo-iconic is applicable, from homo meaning
the same, and icon meaning representation.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Now compare this with the working definition in the above: bullets 1, 2, and 3
are not at all concerned with sameness of internal and external representation.
However, they are concerned with different kinds of sameness: Mostly, the fact
that datastructures in the code, and the program text, are represented in the
same way.&lt;/p&gt;

&lt;h3 id=&quot;conclusions&quot;&gt;Conclusions&lt;/h3&gt;

&lt;p&gt;I’d still argue that homoiconicity is a concept that confuses more than it
clarifies, mostly because of &lt;a href=&quot;/dont-say-homoiconic/#alternate-definitions&quot;&gt;the many competing
definitions&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Anyone who wants to get a point across is better off by simply referring to
more direct properties of a language such as “has a nice literal syntax for
structured data” or “the shape of the AST is immediately apparent from looking
at your screen”.&lt;/p&gt;

&lt;p&gt;Still, faced with continued usage, the working definition above might at least
serve as a dictionary for the perplexed.&lt;/p&gt;


  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/homoiconicity-revisited/&quot;&gt;Homoiconicity revisited&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on June 02, 2020.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Animating history: Implementation]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/animating-implementation/" />
  <id>http://www.expressionsofchange.org/animating-implementation</id>
  <published>2018-10-16T00:00:00+02:00</published>
  <updated>2018-10-16T00:00:00+02:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;h2 id=&quot;what-are-history-animations&quot;&gt;What are history-animations?&lt;/h2&gt;

&lt;p&gt;History-animations build on the following feature (one that was already existing):
whenever we select an s-expression in our “tree” (the structural
view on the right hand side of the window) we show the history of that
particular s-expression in the panel on the left. That is, whenever we change the cursor in
the tree, we switch what is shown in the history.&lt;/p&gt;

&lt;p&gt;The animation under discussion: any part of history that shows up both before
and after this switch will “float” from its pre-switch position to its
post-switch position in a number of steps. The idea is to make it visually more
clear that there is a relationship between the histories at different levels of
the tree.&lt;/p&gt;

&lt;p&gt;More details and examples can be found in a &lt;a href=&quot;/animating-history-transitions/&quot;&gt;separate
article&lt;/a&gt;. In the present article we’ll zoom in
on the implementation.&lt;/p&gt;

&lt;p&gt;The current version of the editor supports 2 versions of rendering histories:
one in which the histories are rendered as s-expressions themselves, as
presented in the paper &lt;a href=&quot;/assets/papers/clef-design.pdf&quot;&gt;the paper “Clef
Design”&lt;/a&gt;; one in which the effects of each note
are shown in the context of the structure on which it is played (i.e.: more
like a traditional rendering of a &lt;em&gt;diff&lt;/em&gt;). Animations of transitions are
implemented for both of these; where the implementations diverge this will be
pointed out in the below.&lt;/p&gt;

&lt;h2 id=&quot;identity-of-notes-and-textures&quot;&gt;Identity of notes and textures&lt;/h2&gt;

&lt;p&gt;The key idea in the animations is to float textures from some pre-switch to a
post-switch location. This hinges on the assumption that we have a shared
identity for the textures pre- and post-switch. E.g. to float some open-bracket
from one location to the next, we need to know which open-bracket we’re talking
about (there are many, and they look very similar).&lt;/p&gt;

&lt;p&gt;Note that the particular animation under consideration is the following: when
swichting which part of our structural view (the “tree”) is selected, update
the historical view.&lt;/p&gt;

&lt;p&gt;Thus, the assumption of shared identity, in this case, is: there is overlap
between the histories of different parts of our tree. For each of the elements
(notes) of the history we can establish an identity, and when viewing a
different history, we can establish whether any two notes across these two
histories are the same one, i.e. share this identity.&lt;/p&gt;

&lt;p&gt;The fact that parts of histories are shared across different parts of our
structure is detailed in &lt;a href=&quot;/assets/papers/clef-design.pdf&quot;&gt;the paper “Clef
Design”&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In terms of the implementation, the solution is to have some addressing scheme
for the textures that is global in the sense that it is shared between the
pre-and post-switch environments.  Using this addressing scheme we can identify
textures: same address means same texture.&lt;/p&gt;

&lt;p&gt;Such an addressing scheme for textures is obtained in a number of steps.&lt;/p&gt;

&lt;h3 id=&quot;noteaddress&quot;&gt;NoteAddress&lt;/h3&gt;

&lt;p&gt;The first step is to annotate each note in the “global history” (the history of
the whole tree) in such a way that we can uniquely identify each note.
&lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/e7a74705c7de/dsn/s_expr/clef_address.py#L154&quot;&gt;Implementation&lt;/a&gt;
and
&lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/e7a74705c7de/widgets/history.py#L161&quot;&gt;calling&lt;/a&gt;
&lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/e7a74705c7de/widgets/ic_history.py#L322&quot;&gt;locations&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The formalization of the note-address is implemented in the class &lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/4e519b03dee1d5e166565519b1c910afdce8d19c/dsn/s_expr/note_address.py#L6&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;NoteAddress&lt;/code&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The intuition here is: when the whole history is written out as an expression,
the address of a particular note is a path trough that expression. An example
could be: of the global score, take the 6th item; of that item take the only
child, of that item again take the only child. The 2 main possible parts of such
paths are: the &lt;em&gt;nth&lt;/em&gt; item of a Score, and the only child. &lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/4e519b03dee1d5e166565519b1c910afdce8d19c/doctests/note_address.txt&quot;&gt;The
doctests&lt;/a&gt;
provide further details.&lt;/p&gt;

&lt;h3 id=&quot;push-global-noteaddress-to-the-tree&quot;&gt;Push global NoteAddress to the tree&lt;/h3&gt;

&lt;p&gt;In the second step, we construct a tree by playing this global history of notes,
annotated with their global address
(&lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/e7a74705c7de/widgets/history.py#L163&quot;&gt;here&lt;/a&gt;
and
&lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/e7a74705c7de/widgets/ic_history.py#L324&quot;&gt;here&lt;/a&gt;).
We use the regular mechanism of playing a score to get a tree (&lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/e7a74705c7de/dsn/s_expr/clef_address.py#L167&quot;&gt;This
one&lt;/a&gt;
– in fact, it’s not 100% identical for implementation reasons, as documented
in the code, but in terms of behavior it is). The only difference is: because the
input Notes have now been annotated with a global address, the scores as
constructed at each sub-expression in the resulting tree are now consisting of
notes which have a global address. This means that when we fetch the “local
score” (the score to be rendered) we have information about the global address
of each note.&lt;/p&gt;

&lt;h3 id=&quot;texture-addresses&quot;&gt;Texture-addresses&lt;/h3&gt;

&lt;p&gt;Finally, we make sure to keep the annotations around in each step of the
conversion to textures, as well as add conversion-specific information when
needed. The implementations of this final step are unique for each of the two
different styles of rendering.&lt;/p&gt;

&lt;h4 id=&quot;els18-style-rendering&quot;&gt;ELS’18 style rendering&lt;/h4&gt;

&lt;p&gt;In the case rendering of in the style of the ELS’18 paper, the tree of notes is
first converted to an s-expr, and these s-expressions are then converted to the
actual textures with locations.&lt;/p&gt;

&lt;p&gt;We need step-specific address information for each of these steps. When converting to
an s-expression, we annotate the elements that are specific to the fact that the
note is being rendered as an s-expression (i.e. the fact that the Note’s fields
and its type, when converted to an s-expression, turn into particular
further s-expressions). &lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/e7a74705c7de/dsn/s_expr/clef_address.py#L31&quot;&gt;Let’s consider the case of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;become-atom&lt;/code&gt; as an
example&lt;/a&gt;:
when the note &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(become-atom foo)&lt;/code&gt; is represented as an s-expression the whole
s-expression is annotated as representing the whole note (by not providing any
further annotation), the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;become-atom&lt;/code&gt; is annotated as being the name of
the note, and the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;foo&lt;/code&gt; is annotated as being the field &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;atom&lt;/code&gt; of that
note.&lt;/p&gt;

&lt;p&gt;When converting these s-expressions to textures similar further annotations
are necessary. For example: a list-expression is rendered as 2 textures, one
for &lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/e7a74705c7de/widgets/history.py#L348&quot;&gt;each&lt;/a&gt; &lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/e7a74705c7de/widgets/history.py#L361&quot;&gt;bracket&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A particular property of this style of rendering histories, is that the
recursive nature of the histories is preserved in the rendering. That is: a note
may contain further notes; when the note is rendered, the notes it contains are
also rendered.&lt;/p&gt;

&lt;p&gt;With regards to the assignment of addresses to textures, the implication is
straightforward: each rendered note is assigned with the address of that
particular note.&lt;/p&gt;

&lt;p&gt;An example is drawn below: if the chord below is the item at position 1 in some
other history, the children of that chord are at some subpath.&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;(chord ((insert 0 (become-list)) (extend 0 (insert 0 (become-list)))))
^       ^                        ^
|       |                        |
(@1) (@1, @0)                 (@1, @1)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The effect of this approach on the animation is precisely as intended: when
switching from a larger context to a smaller one, the “surrounding” notes that
are not applicable in the smaller context float out of view; but those that are
applicable in both views (the inner ones), float from their old position on the
screen to the new one. (The reverse applies when switching from a smaller
context to one surrounding it)&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/movies/transitions-clef-design.gif&quot; alt=&quot;Surrounding context disappears.&quot; /&gt;&lt;/p&gt;

&lt;h4 id=&quot;ic-history&quot;&gt;IC History&lt;/h4&gt;

&lt;p&gt;Another way of rendering notes is by rendering them “in their structural
context”. That is: by showing their effect on the existing structure on which
they are being played. This is how &lt;em&gt;diffs&lt;/em&gt; are traditionally displayed.&lt;/p&gt;

&lt;p&gt;In this view, the recursive nature of notes is not made explicit.  For each note
in some list of notes (for example: those that make up a single score), the
effect of each indivual note on a structure are grouped together. The fact each
such note may itself be composed of any number of other notes is left implict.&lt;/p&gt;

&lt;p&gt;Thus, when switching from a larger historical context to a smaller one, it is
not the case that some surrounding &lt;em&gt;notes&lt;/em&gt; disappear, while notes contained by
them remain in view.&lt;/p&gt;

&lt;p&gt;There simply is no direct rendering of notes in this view: everything that is
rendered is a structure and some effects on that structure. This means that any
addressing must also apply to such structures. And that any floating of related
elements is always floating of some structural element.&lt;/p&gt;

&lt;p&gt;It is at this structural level that a similar effect as in the above, of
surrounding context disappearing, can be seen: when switching to a smaller
structural context, less surrounding structure is shown in the in-context
rendering of history, and vise versa for switching to a larger, surrounding,
context:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/movies/transition-9.gif&quot; alt=&quot;Transition of &apos;9&apos;.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;The implentation details are in the implementing class,
&lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/4e519b03dee1/dsn/s_expr/in_context_display.py#L45&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ICHAddress&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The mixing of ‘construction’ and ‘structure’ is reflected in the address of the
rendered elements; each rendered element is denoted first by the note which it
represents (in terms of a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;NoteAddress&lt;/code&gt;), and second by an address (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;t_address&lt;/code&gt;,
for stability over time) in the tree. (further steps in the rendering chain add
further details, i.e.  &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;icd_specific&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;render_specific&lt;/code&gt;)&lt;/p&gt;

&lt;p&gt;One final caveat: the NoteAddress &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;NoteAddress&lt;/code&gt; part of this &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ICHAddress&lt;/code&gt; is
always the address of the &lt;a href=&quot;https://github.com/expressionsofchange/nerf1/blob/4e519b03dee1/widgets/ic_history.py#L91&quot;&gt;deepest&lt;/a&gt; (leaf-most) possible note. For example, when
rendering the note &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(extend 0 (insert 0 (become-list)))&lt;/code&gt;, the address of
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(become-list)&lt;/code&gt; is used in the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ICHAddress&lt;/code&gt;.  This ensures we have a singular
identity across context-switches. (It is only this deepest NoteAddress that can
be relied on to always be availalble).&lt;/p&gt;

&lt;h3 id=&quot;the-animation&quot;&gt;The animation&lt;/h3&gt;

&lt;p&gt;The actual animation is rather straightforward: do a linear interpolation for
(source, target) for the attributes (x, y, alpha).&lt;/p&gt;

&lt;p&gt;We set a clock at an interval (I’ve set 1/60, but I’m not actually getting this
&lt;em&gt;at all&lt;/em&gt; on my local machine).  Kivy will tell you how much time has actually
passed since the last tick. We then calculate the fraction &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;dt / remaining_time&lt;/code&gt;.
This approach is automatically robust for missed frames (i.e. the missed frame
will not be rendered, but the total animation time and the position of the
texture at the next frame are unaffected)&lt;/p&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/animating-implementation/&quot;&gt;Animating history: Implementation&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on October 16, 2018.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Animating history transitions]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/animating-history-transitions/" />
  <id>http://www.expressionsofchange.org/animating-history-transitions</id>
  <published>2018-06-01T00:00:00+02:00</published>
  <updated>2018-06-01T00:00:00+02:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;In &lt;em&gt;Expressions of Change&lt;/em&gt; we take modifications to programs to be the primary building block of program construction. We do this in the expectation that that the availability of well-structured historic information across our toolchain will prove invaluable when facing the typical challenges of program modification.&lt;/p&gt;

&lt;p&gt;One of the first such benefits is the ability to inspect history at any level of program granularity as it pertains to that level. That is: if a program consists of e.g. modules, classes, functions, statements and expressions, to be able to inspect the history of each of these, and to see how these histories relate.&lt;/p&gt;

&lt;p&gt;Over the past few weeks I’ve been working on a way to visualize this central idea, under the working titles of “animations” or “transitions”. Here you can see it in action:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/movies/transitions-in-context.gif&quot; alt=&quot;Full demo with &amp;quot;in context&amp;quot; historical view.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;How should we understand the two panels shown above? The panel on the right represents program &lt;em&gt;structure&lt;/em&gt;, i.e. what we usually think of as “the program”. The structure under consideration is an &lt;a href=&quot;/introducing-the-editor/#structure-s-expressions&quot;&gt;s-expression&lt;/a&gt;, recursively defined as either a list-expression of further s-expressions between parentheses &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;)&lt;/code&gt;, or an atom.&lt;/p&gt;

&lt;p&gt;The panel on the left represents a historic overview of the program’s &lt;em&gt;construction&lt;/em&gt;: something we usually think of as “Version Management”, although in &lt;em&gt;Expressions of Change&lt;/em&gt; we avoid that term, because we view the history of the program as at least equally as important as the program itself.&lt;/p&gt;

&lt;p&gt;In the above, the representation of construction is “in its structural context”. That is: each line shows how a certain modification affects the existing structure: parts of the structure that already existed before the modification are shown in grey, parts that are added by the modification are shown in black.&lt;sup id=&quot;fnref:not-diff&quot;&gt;&lt;a href=&quot;#fn:not-diff&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; (Deletions would be shown in red, but the example above contains no deletions). The single line with inverted colors denotes the cursor in the historic view.&lt;/p&gt;

&lt;p&gt;Importantly, the two panels are connected: whenever we select some &lt;nobr&gt;s-expression&lt;/nobr&gt; in the structural view on the right, the panel on the left follows suit, and shows the history of that structure.&lt;sup id=&quot;fnref:always-follow&quot;&gt;&lt;a href=&quot;#fn:always-follow&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;In the above, we select the expression &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(+ (* 6 9) 12)&lt;/code&gt;, its sub-expression &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(* 6 9))&lt;/code&gt; and the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;9&lt;/code&gt; respectively, each time showing the relevant histories.&lt;/p&gt;

&lt;h3 id=&quot;animations--transitions&quot;&gt;Animations / transitions&lt;/h3&gt;

&lt;p&gt;The innovation from the past few weeks is that, whenever we switch between the display of one history and another, we show the relationship between the two using an animation.&lt;/p&gt;

&lt;p&gt;The simplest example of such a relationship is the one between the history of the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;9&lt;/code&gt; (a trivial history, consisting only of its creation) and the history of the whole expression &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(+ (* 6 9) 12)&lt;/code&gt;. When we switch from the former to the latter, we see two additional types of information: first, we see that the creation of the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;9&lt;/code&gt; is preceded by the construction of the rest of the expression beginning with the creation of the list &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;()&lt;/code&gt; and ending with the addition of the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;12&lt;/code&gt;. Second, we see where it fits into the greater expression, i.e. to the right of the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;6&lt;/code&gt; and left of a closing bracket. Here this single transition is singled out:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/movies/transition-9.gif&quot; alt=&quot;Transition of &apos;9&apos;.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;The transition between the histories of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(+ (* 6 9) 12)&lt;/code&gt; and the sub-expression &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(* 6 9))&lt;/code&gt; forms a more exiting example: we can see how each of the 4 elements of the latter relate to the history of the former, and how they fit into a larger context. Again, we show only the single transition here:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/movies/transition-star-6-9.gif&quot; alt=&quot;Transition of &apos;(* 6 9)&apos;.&quot; /&gt;&lt;/p&gt;

&lt;h3 id=&quot;why-bother&quot;&gt;Why bother?&lt;/h3&gt;

&lt;p&gt;I believe these animations are useful for two separate reasons. First, when explaining the concepts of &lt;em&gt;Expressions of Change&lt;/em&gt;, a (moving) picture says more than a 1000 words. In this case, the point that’s being explained visually is the idea of “history at any level”.&lt;/p&gt;

&lt;p&gt;Second, in an editor for actual use, the animations serve as a visual aid: transitioning smoothly from one history to the next really helps you keep your bearings while navigating.&lt;/p&gt;

&lt;p&gt;By the way, the animations in the &lt;em&gt;gifs&lt;/em&gt; shown in the above are slowed down to a full 1.5 seconds for both dramatic effect and to ease understanding of what’s happening in the context of a blog-post. In an actual edit environment we’ll want to balance between too fast and too slow: the former makes it impossible to see what’s going on, the latter leads to annoying waits. I found personally found 0.5 seconds per animation to be a good trade-off:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/movies/normal-speed.gif&quot; alt=&quot;Transitions at normal speed.&quot; /&gt;&lt;/p&gt;

&lt;h3 id=&quot;explicit-notes&quot;&gt;“Explicit” notes&lt;/h3&gt;

&lt;p&gt;The example animations shown in this blog-post correspond directly with the main example from the paper &lt;a href=&quot;/assets/papers/clef-design.pdf&quot;&gt;“Clef design”&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;In that paper, we used a more explicit notation for the modifications (called “notes”): those were themselves modelled as s-expressions, rather than displayed by showing how they affect some structure.&lt;sup id=&quot;fnref:notational-difference&quot;&gt;&lt;a href=&quot;#fn:notational-difference&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;3&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;If we apply the ideas of this blog-post to the notation from the paper, it looks like this:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/movies/transitions-clef-design.gif&quot; alt=&quot;Full demo with &amp;quot;Clef Design&amp;quot; historical view.&quot; /&gt;&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:not-diff&quot;&gt;
      &lt;p&gt;One might even be tempted to call this a “diff”, but that would suggest a primacy of &lt;em&gt;structure&lt;/em&gt; rather than &lt;em&gt;construction&lt;/em&gt;, i.e. it somewhat implies there are two structures first, and we simply calculate the difference. &lt;a href=&quot;#fnref:not-diff&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:always-follow&quot;&gt;
      &lt;p&gt;In the above the view of construction always follows the structure’s cursor. In practical situations it’s probably useful to be able to toggle this following of the cursor on or off, retaining the ability to see the history of sub-expressions in their larger context as needed. &lt;a href=&quot;#fnref:always-follow&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:notational-difference&quot;&gt;
      &lt;p&gt;In fact, the &lt;em&gt;Clef&lt;/em&gt; in the demonstrations above differs from the &lt;em&gt;Clef&lt;/em&gt; in the paper in one way: &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(insert  ...&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(delete ...&lt;/code&gt; taking a single note, rather than a score, as an argument. &lt;a href=&quot;#fnref:notational-difference&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/animating-history-transitions/&quot;&gt;Animating history transitions&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on June 01, 2018.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Source code published]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/source-code-published/" />
  <id>http://www.expressionsofchange.org/source-code-published</id>
  <published>2018-05-05T00:00:00+02:00</published>
  <updated>2018-05-05T00:00:00+02:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;We can distinguish 2 projects, “&lt;a href=&quot;https://github.com/expressionsofchange/nerf0&quot;&gt;nerf0&lt;/a&gt;” and “&lt;a href=&quot;https://github.com/expressionsofchange/nerf1&quot;&gt;nerf1&lt;/a&gt;”&lt;sup id=&quot;fnref:nerf-explained&quot;&gt;&lt;a href=&quot;#fn:nerf-explained&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;The brief summary of which project to refer to is:&lt;/p&gt;

&lt;h3 id=&quot;nerf-0&quot;&gt;Nerf 0:&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;Static Analysis (first steps)&lt;/li&gt;
  &lt;li&gt;mini-interpreter&lt;/li&gt;
  &lt;li&gt;Alternative Clef, in which a note &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; may replace a given node with one that can be constructed from an arbitrary history (not just a history that’s the result of extending the given history) – this approach also has consequences on how undo may be modelled.&lt;/li&gt;
  &lt;li&gt;merging (“weaving”) of 2 histories (first steps only)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;nerf-1&quot;&gt;Nerf 1:&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;the most up to date Clef.&lt;/li&gt;
  &lt;li&gt;work on visualising modifications in a more human friendly way&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Nerf1 was created by “scavenging” code from nerf0; The primary goal: to adhere much more closely to the Clef that is presented in the paper “Clef Design”, as presented on ELS ‘18. (In fact, the Clef in this project differs from the presented Clef in one important way: &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert&lt;/code&gt; &amp;amp; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Delete&lt;/code&gt; take a single note, rather than a score, as an argument) This is the program that was used in the demo for the presentation at ELS.&lt;/p&gt;

&lt;p&gt;Both projects should be seen as sketches, as bases for experiments, more so than as a finished product that is in any sense ready for production.&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:nerf-explained&quot;&gt;
      &lt;p&gt;“nerf”, the Dutch word for wood grain, is yet another metaphore of how the mechanism of growth has a direct effect on the grown product. &lt;a href=&quot;#fnref:nerf-explained&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/source-code-published/&quot;&gt;Source code published&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on May 05, 2018.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Talk at the European Lisp Symposium]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/clef-design-marbella-talk-live/" />
  <id>http://www.expressionsofchange.org/clef-design-marbella-talk-live</id>
  <published>2018-04-19T00:00:00+02:00</published>
  <updated>2018-04-19T00:00:00+02:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;The video of the talk at The European Lisp Symposium is live.&lt;/p&gt;

&lt;p class=&quot;videoWrapper&quot;&gt;
&lt;iframe width=&quot;680&quot; height=&quot;382&quot; src=&quot;https://www.youtube.com/embed/qHVrKQvFODI&quot; frameborder=&quot;0&quot; allowfullscreen=&quot;&quot;&gt;&lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;The associated paper may be downloaded using the link below:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;/assets/papers/clef-design.pdf&quot;&gt;Clef design: Thoughts on the Formalization of Program Construction&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The relevant bibtex entry may be found below.&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;@INPROCEEDINGS{VanSchelven2018,
    author = &quot;Klaas van Schelven&quot;,
    pages = {&quot;94-101&quot;},
    title = {Clef Design, Thoughts on the Formalization of Program Construction},
    booktitle = {Proceedings of the 11th European Lisp Symposium},
    address = {Marbella, Spain},
    isbn = {978-1-4503-5183-6},
    year = {2018},
    doi = {10.5281/zenodo.3263960}
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/clef-design-marbella-talk-live/&quot;&gt;Talk at the European Lisp Symposium&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on April 19, 2018.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[ELS 2018: Paper accepted]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/european-lisp-symposium-paper-accepted/" />
  <id>http://www.expressionsofchange.org/european-lisp-symposium-paper-accepted</id>
  <published>2018-03-20T00:00:00+01:00</published>
  <updated>2018-03-20T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;The European Lisp Symposium takes place on April 16th - April 17th in Marbella, Spain.&lt;/p&gt;

&lt;p&gt;The paper may be downloaded using the link below:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;/assets/papers/clef-design.pdf&quot;&gt;Clef design: Thoughts on the Formalization of Program Construction&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The relevant bibtex entry may be found below.&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;@INPROCEEDINGS{VanSchelven2018,
    author = &quot;Klaas van Schelven&quot;,
    pages = {&quot;94-101&quot;},
    title = {Clef Design, Thoughts on the Formalization of Program Construction},
    booktitle = {Proceedings of the 11th European Lisp Symposium},
    address = {Marbella, Spain},
    isbn = {978-1-4503-5183-6},
    year = {2018},
    doi = {10.5281/zenodo.3263960}
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/european-lisp-symposium-paper-accepted/&quot;&gt;ELS 2018: Paper accepted&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on March 20, 2018.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Talk at Clojure Meetup]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/clojure-meetup/" />
  <id>http://www.expressionsofchange.org/clojure-meetup</id>
  <published>2018-03-14T00:00:00+01:00</published>
  <updated>2018-03-14T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;The audience was very much involved, which led to some interesting discussions.&lt;/p&gt;

&lt;p&gt;Unfortunately, no recording of the screen was made while giving the demo; the resulting resolution is probably not good enough to follow along; I’ve kept this section in anyway to preserve the discussion with the audience.&lt;/p&gt;

&lt;p&gt;A video of this event is posted below.&lt;/p&gt;

&lt;p class=&quot;videoWrapper&quot;&gt;
&lt;iframe width=&quot;680&quot; height=&quot;382&quot; src=&quot;https://www.youtube.com/embed/pFJ4-fTmCgU&quot; frameborder=&quot;0&quot; allowfullscreen=&quot;&quot;&gt;&lt;/iframe&gt;
&lt;/p&gt;


  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/clojure-meetup/&quot;&gt;Talk at Clojure Meetup&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on March 14, 2018.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Don't say “Homoiconic”]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/dont-say-homoiconic/" />
  <id>http://www.expressionsofchange.org/dont-say-homoiconic</id>
  <published>2018-03-01T00:00:00+01:00</published>
  <updated>2018-03-01T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;What is homoiconicity then? Typical definitions state that it is simply “code as data”, will point to a relationship between a program’s structure and syntax or note that the program source is expressed in a primitive data-type of the language. In the below, we will show that none of these definitions make much sense.&lt;/p&gt;

&lt;p&gt;Before that, however, we’ll return to the original definition, to ensure we have at least some sensible frame of reference for the rest of the article. This is also the definition we’ll return to in the final section, when we put it to the test by examining the canonical example of homoiconicity.&lt;/p&gt;

&lt;h2 id=&quot;historic-definition&quot;&gt;Historic definition&lt;/h2&gt;

&lt;p&gt;The term homoiconic was first coined by Calvin Mooers in the article &lt;a href=&quot;https://dl.acm.org/citation.cfm?doid=800197.806048&quot;&gt;TRAC, A Text-Handling Language&lt;/a&gt; (L. Peter Deutsch was the other author of that article, but he has made it clear to me that Mooers was the originator of the term and Deutsch’s contribution was only on the technical side).&lt;sup id=&quot;fnref:trac-footnote&quot;&gt;&lt;a href=&quot;#fn:trac-footnote&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; In the original article, the meaning of homoiconicity seems to be spelled out quite clearly. The article opens by stating that the TRAC language is homoiconic, although without yet using that term explicitly:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The external and internal forms of the TRAC language are the same.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Before introducing the term itself, the importance of a single representation for viewing and manipulation by the user and interpretation by the computer is repeated no less than 5 times; numbered here in square brackets for clarity:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;One of the main design goals was [1] that the input script of TRAC (what is typed in by the user) should be identical to the text which guides the internal action of the TRAC processor. In other words, [2] TRAC procedures should be stored in memory as a string of characters exactly as the user typed them at the keyboard. [3] If the TRAC procedures themselves evolve new procedures, these new procedures should also be stated in the same script. [..] [4] At any time, it should be possible to display program or procedural information in the same form as the TRAC processor will act upon it during its execution. [5] It is desirable that the internal character code representation be identical to, or very similar to, the external code representation.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This paragraph finally concludes with the definition itself – seemingly leaving no room for doubt:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Because TRAC procedures and text have the same representation inside and outside the processor, the term &lt;em&gt;homo-iconic is&lt;/em&gt; applicable, from &lt;em&gt;homo&lt;/em&gt; meaning the same, and &lt;em&gt;icon&lt;/em&gt; meaning representation.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;alternate-definitions&quot;&gt;Alternate definitions&lt;/h2&gt;

&lt;p&gt;Unfortunately, this rather straightforward definition is not the only one in active use. There is in fact a proliferation of alternative, and entirely misguided, definitions, some of which are quite persistent. It is worth pointing out explicitly why each of them is misguided: if homoiconicity is to have any meaning at all, it’s certainly a good idea to point out what it doesn’t mean.&lt;/p&gt;

&lt;h3 id=&quot;code-as-data&quot;&gt;“Code as data”&lt;/h3&gt;

&lt;p&gt;First, any attempt to explain homoiconicity by using the phrase “Code as data” is quite meaningless. The reason is simply that “Code as data” can be used to indicate any of a large number of quite distinct ideas, some of which are listed below:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Functions as a first-class citizen.&lt;/li&gt;
  &lt;li&gt;The fact that a computer in a Von Neumann architecture stores programs and data in the same memory device.&lt;/li&gt;
  &lt;li&gt;Reflection and metaprogramming.&lt;/li&gt;
  &lt;li&gt;Homoiconicity.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The punchline is in the last bullet in the list: if homoiconicity is “code as data”, and “code as data” is homoiconicity, we’ve simply created a circular definition and explained nothing at all – but with extra potential for confusion because “code as data” may also refer to other ideas.&lt;/p&gt;

&lt;h3 id=&quot;program-structure--syntax&quot;&gt;Program structure &amp;amp; syntax&lt;/h3&gt;

&lt;p&gt;The second definition attempts to draw a connection between the program’s structure and its syntax. For example, there’s this formulation which is currently featuring on &lt;a href=&quot;https://en.wikipedia.org/wiki/Homoiconicity&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Homoiconicity [..] is a property [..] in which the program structure is similar to its syntax&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is a category error, that is, a similarity is drawn between ideas that exist at two separate levels of abstraction.&lt;/p&gt;

&lt;p&gt;Syntax makes a statement &lt;em&gt;about&lt;/em&gt; program structure, that is the syntax of a language is the set of rules that defines the combinations of symbols that are considered to be a correctly structured program.&lt;/p&gt;

&lt;p&gt;To say that the rules defining the program’s structure are similar to the program’s structure makes no sense. First, since a single syntax defines a potentially infinite set of correctly structured programs, which program are we talking about as a reference for comparison? Second, what would the measure of similarity across these 2 levels of abstraction be?&lt;/p&gt;

&lt;p&gt;Similar confusion surrounding the terminology of syntax and structure can be found further down in the same article:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;If a language is homoiconic, it means that the language text has the same structure as its abstract syntax tree (AST)&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is also wrong, but for the opposite reason: the abstract syntax tree is by definition a representation of the structure of the language text. This is the case for any AST, in any language. Thus, to say that this has anything to do with homoiconicity is quite meaningless.&lt;/p&gt;

&lt;h3 id=&quot;meta-circular-evaluator&quot;&gt;Meta-circular evaluator&lt;/h3&gt;

&lt;p&gt;Third, the property of homoiconicity is often conflated with the existence of a meta-circular evaluator, but the two concepts are entirely separate. An example of such confusion is the following quote from the Wikipedia article:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;A typical demonstration of homoiconicity is the meta-circular evaluator.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;To understand this claim, let’s first examine what a meta-circular evaluator is. The term was coined originally in 1972 by John C. Reynolds in his paper &lt;a href=&quot;http://people.cs.uchicago.edu/~blume/classes/aut2008/proglang/papers/definterp.pdf&quot;&gt;Definitional Interpreters for Higher-Order Programming Languages&lt;/a&gt;.&lt;sup id=&quot;fnref:sicp-meta-circular&quot;&gt;&lt;a href=&quot;#fn:sicp-meta-circular&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;We have coined the word “meta-circular” to indicate the basic character of this interpreter: It defines each feature of the defined language by using the corresponding feature of the defining language. For example, when eval is applied to an application expression [..] of the defined language, it evaluates an application expression [..] in the defining language.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The relationship with homoiconicity, if it exists, is accidental at most.&lt;/p&gt;

&lt;p&gt;It is quite possible to construct a meta-circular interpreter in languages which are clearly not homoiconic. Consider for example a Python-interpreter, written in Python, which implements each language-construct from Python in Python: this satisfies the definition of a meta-circular evaluator. However, Python is not homoiconic, because the internal and external forms of the language are quite different: the external representation of Python programs is the text file, and the internal form, on which the interpreter operates, is bytecode.&lt;/p&gt;

&lt;h3 id=&quot;primitive-data-types&quot;&gt;Primitive data types&lt;/h3&gt;

&lt;p&gt;Another often seen definition of homoiconicity focuses on the relationship between the &lt;em&gt;primitive data types&lt;/em&gt; in the language and the representation of the language. Again, we quote from Wikipedia:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;In a homoiconic language, the primary representation of programs is also a data structure in a primitive type of the language itself.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Again, this is a definition which doesn’t hold up to any scrutiny.&lt;/p&gt;

&lt;p&gt;First, consider that most languages have a primitive datatype for strings of text, and the programs in most languages are presented to the user as a string of text. Should we conclude from this that most languages are homoiconic?&lt;/p&gt;

&lt;p&gt;Second, for most languages, it is possible to write a parser in that language itself, that stores the resulting abstract syntax tree in a primitive type in the language. For example: we can write a parser in &lt;em&gt;Java&lt;/em&gt; that parses Java source code into Java objects, which are primitive data types in the language. Should we conclude from this that Java is homoiconic?&lt;/p&gt;

&lt;p&gt;In both cases, the answer is clearly “no” – if all languages are homoiconic, the terms loses all meaning.&lt;/p&gt;

&lt;h2 id=&quot;an-example&quot;&gt;An example&lt;/h2&gt;

&lt;p&gt;Now we know what homoiconicity is not, the way is cleared for a discussion on what it is. Let’s return to the original definition, and try to fit this to an example. Remember, the original definition of homoiconicity centers on &lt;em&gt;a similarity between the internal and external representations of a language&lt;/em&gt;. In the literature, Lisps are the favorite example of homoiconicity; we will stick with that tradition here, and examine the case of an arbitrary Lisp program in an arbitray language in the Lisp family.&lt;sup id=&quot;fnref:lisp-not-homoiconic&quot;&gt;&lt;a href=&quot;#fn:lisp-not-homoiconic&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;3&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;First, the external representation: the basic syntactical element of Lisps is the s-expression. Thus, a typical program in Lisp is represented to the programmer as nothing more than an s-expression.&lt;/p&gt;

&lt;p&gt;Second, the internal representation. The evaluation of a Lisp program is typically defined in terms of those s-expressions directly. One example is the definition of Scheme given by &lt;a href=&quot;https://mitpress.mit.edu/sites/default/files/6515.pdf&quot;&gt;Abelson &amp;amp; Sussman&lt;/a&gt;, another is the definition &lt;a href=&quot;/l-a-toy-language/#case-analysis&quot;&gt;given on this site&lt;/a&gt;. In both cases, the semantics of the language are given in terms of a case-analysis on an s-expression. Thus, the (virtual) machine operates on s-expressions directly, and we can say the internal representation of the program is an s-expression.&lt;/p&gt;

&lt;p&gt;Because the internal and external representations are the same, we might say that this combination of language and interpreter is homoiconic.&lt;/p&gt;

&lt;h3 id=&quot;objections&quot;&gt;Objections&lt;/h3&gt;

&lt;p&gt;The observant reader may raise at least two objections against the above.&lt;/p&gt;

&lt;p&gt;The first objection concerns the external representation. In the above we simply stated that the external representation is an s-expression.  In most practical programming environments, however, the actual representation of program sources is as text files which contain strings of characters. It is only after parsing this text that the representation is really an s-expression. In other words: in practical environments the external representation is not an s-expression, but text.&lt;/p&gt;

&lt;p&gt;The second objection concerns the internal representation. Practical implementations of Lisp interpreters do generally not operate actually directly on s-expressions internally for performance reasons. Even though a Lisp might be defined in terms of a case-analysis on s-expressions, it is not usually implemented as such. Thus, the internal representation is not actually an s-expression in practice.&lt;/p&gt;

&lt;h3 id=&quot;para-iconic&quot;&gt;Para-iconic&lt;/h3&gt;

&lt;p&gt;In an attempt to counter these arguments, we might slightly alter our definition, to state that homoiconicity is not a boolean property, but a scalar one: a language in a given environment can be homoiconic to some degree.&lt;/p&gt;

&lt;p&gt;In fact, the original definition by Mooers and Deutsch leaves some space for such an interpretation when it states that “the internal [..] representation be identical to, &lt;em&gt;or very similar to&lt;/em&gt;, the external [..] representation.” (emphasis mine).&lt;/p&gt;

&lt;p&gt;A more precise term for such “partial homoiconicity”, could perhaps be para-iconicity or simula-iconicity. However, that would be introducing yet another term into an already confused vocabulary. In any case, given this scalar interpretation, we can see that the example above is indeed homoiconic to a very large degree:&lt;/p&gt;

&lt;p&gt;Regarding the external representation: parsing of s-expressions is trivial, as is the reverse operation of converting s-expressions to some printable form. The use of explicit brackets for all list-expressions ensures that the structure of s-expressions is always explicitly visible. Thus, to say that the external representation of a Lisp program is an s-expression holds at least some truth.&lt;/p&gt;

&lt;p&gt;Regarding the internal representation: even though an actual Lisp interpreter might be optimized, its operational semantics are defined in terms of a virtual machine that operates on s-expressions. Any properly implemented optimization will have the same behavior (barring performance) as its non-optimized counterpart. Thus, from the perspective of the programmer the internal representation is still the s-expression.&lt;sup id=&quot;fnref:what-is-internal&quot;&gt;&lt;a href=&quot;#fn:what-is-internal&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;4&lt;/a&gt;&lt;/sup&gt; Thus, to say that the internal representation is an s-expression holds some truth as well.&lt;/p&gt;

&lt;p&gt;For comparison with a “less homoiconic” language, we could consider the example of compiled &lt;em&gt;C&lt;/em&gt;: the external representation of a &lt;em&gt;C&lt;/em&gt; program is as a piece of program text; the internal representation is in the form of machine code. Strings of text and machine code are quite dissimilar, and to map between them takes considerable effort.&lt;/p&gt;

&lt;h2 id=&quot;conclusions&quot;&gt;Conclusions&lt;/h2&gt;

&lt;p&gt;Homoiconicity is a term surrounded by much confusion.&lt;/p&gt;

&lt;p&gt;Some of this confusion can be cleared up, by putting aside misguided definitions and returning to the original definition: languages which have the same external and internal representations are homoiconic.&lt;/p&gt;

&lt;p&gt;Unfortunately, that doesn’t resolve the issue entirely, because even when sticking to a seemingly sensible definition, we can see a lot of problems with the term: it is almost never the case that the external representation and the internal representation are exactly the same.&lt;/p&gt;

&lt;p&gt;We are forced to withdraw to a position of “para-iconicity”: the idea that homoiconicity is a scalar property rather than a boolean one. In that view, a language which has at least some similarity between &lt;em&gt;some&lt;/em&gt; external representation and &lt;em&gt;some model&lt;/em&gt; for the internal representation could be said to be homoiconic.&lt;/p&gt;

&lt;p&gt;However, that’s not a very good position to be in. I’d say it’s much easier to just talk about certain properties of your favorite language directly. For example, “the syntax is easy to parse”, or “the semantics of the language are completely defined on the single page of a book”. That is, there is much to be said in favor of Lisp’s s-expressions without resorting to use such a poorly defined term.&lt;/p&gt;

&lt;p&gt;Maybe it’s time to stop saying “homoiconic”.&lt;/p&gt;

&lt;h3 id=&quot;afterword-l-peter-deutsch-respondsh3&quot;&gt;Afterword: L. Peter Deutsch responds&amp;lt;/h3&amp;gt;&lt;/h3&gt;

&lt;p&gt;In June 2020, I reached out to L. Peter Deutsch with the request to read and comment on the above. He wrote the following:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;I did not coin this word [homoiconic] – it was 100% the creation of my friend the late
Calvin Mooers, who was the original conceiver of TRAC.  My contribution in
working with Calvin was to take his concept, turn it into a rigorous
language definition, and implement it.&lt;/p&gt;

  &lt;p&gt;Having read through your blog article, I would say that I essentially agree
with your argument.  In my view, the root “icon” refers specifically to the
&lt;em&gt;concrete appearance&lt;/em&gt; of something.  For example, “WYSIWYG” in the context
of text editing is arguably the same idea as “homoiconic” in the context of
programming languages, contrasted to languages like TeX where the document
has two very different appearances (the TeX input file and the output
document).  WYSIWYG editors are homo-iconic not simply in the &lt;em&gt;characters&lt;/em&gt;
of the document, but also in their &lt;em&gt;formatting&lt;/em&gt; (font, spacing, …); but I
think the same concept applies.&lt;/p&gt;

  &lt;p&gt;There are other homoiconic macro languages: while TRAC may be the earliest
on record (I don’t remember the date or specifications of Strachey’s GPM),
m4, which I use actively as a preprocessor for other languages, also fits
the concept.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:trac-footnote&quot;&gt;
      &lt;p&gt;A footnote in that article points out that this was “following suggestion of McCullough W.S., based upon terminology due to Peirce, C.S.” It appears that the “Peirce” in question is Charles Sander Peirce, who wrote extensively on &lt;em&gt;semiotics&lt;/em&gt;, which the study of signs, although it is not clear whether and where he used the phrase homo-iconic explicitly. W.S. McCulloch is likely to be Warren Sturgis McCulloch, who was in Cambridge at the time. &lt;a href=&quot;#fnref:trac-footnote&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:sicp-meta-circular&quot;&gt;
      &lt;p&gt;The definition in the influential &lt;a href=&quot;https://mitpress.mit.edu/sites/default/files/6515.pdf&quot;&gt;Structure and Interpretation of Computer Programs&lt;/a&gt;, differs somewhat: “An evaluator that is written in the same language that it evaluates is said to be metacircular.” Although the constraint that each feature of the defined language is implemented in using the corresponding feature of the defined language is not repeated in that definition, the metacircular evaluator as defined in the book satisfies it in practice. &lt;a href=&quot;#fnref:sicp-meta-circular&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:lisp-not-homoiconic&quot;&gt;
      &lt;p&gt;In fact, the original article &lt;em&gt;excludes&lt;/em&gt; Lisp as an example of homoiconicity, but only because Lisp had not settled on a single representation in terms of s-expressions at the time of writing: “Finally, LISP is troubled with a dual language problem: an M-language, which is easy to read, and is used externally, and an S-language, with which the LISP processor operates, and which is usable externally only by the hardened initiates.  It should be noted here that were the S-language the only LISP language, LISP would be close to being homo-iconic (excluding the machine-language functions).” &lt;a href=&quot;#fnref:lisp-not-homoiconic&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:what-is-internal&quot;&gt;
      &lt;p&gt;Taking this view to its ultimate consequence, however, raises even further questions around the concept of homoiconicity: for a well-encapsulated machine, we cannot observe its inner workings &lt;em&gt;by definition&lt;/em&gt;; in that view, making any statement about the internal representation of the machine is meaningless. More generally, the original definition has the problem that the idea that there is a single external and a single internal representation of the program does not match with reality. In fact, there is a whole chain of representations, including electrons in the brain of the programmer, photons emitted from the screen, program text, machine code, and electrons moving in the CPU. &lt;a href=&quot;#fnref:what-is-internal&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/dont-say-homoiconic/&quot;&gt;Don't say “Homoiconic”&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on March 01, 2018.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Lambda Days 2018]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/lambda-days-2018-lightning-talk/" />
  <id>http://www.expressionsofchange.org/lambda-days-2018-lightning-talk</id>
  <published>2018-02-26T00:00:00+01:00</published>
  <updated>2018-02-26T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;&lt;a href=&quot;http://www.lambdadays.org/lambdadays2018&quot;&gt;Lambda Days&lt;/a&gt; is a “one of a kind experience in the functional world.”&lt;/p&gt;

&lt;p&gt;It was held on 22 &amp;amp; 24 February 2018 in Kraków, Poland.&lt;/p&gt;

&lt;p class=&quot;videoWrapper&quot;&gt;
&lt;iframe width=&quot;680&quot; height=&quot;382&quot; src=&quot;https://www.youtube.com/embed/KA2TjX595tA?cc_load_policy=1&amp;amp;cc_lang_pref=en&quot; frameborder=&quot;0&quot; allowfullscreen=&quot;&quot;&gt;&lt;/iframe&gt;
&lt;/p&gt;


  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/lambda-days-2018-lightning-talk/&quot;&gt;Lambda Days 2018&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on February 26, 2018.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Expressions of Change in under 15 minutes]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/expressions-of-change-in-under-15-minutes/" />
  <id>http://www.expressionsofchange.org/expressions-of-change-in-under-15-minutes</id>
  <published>2018-02-20T00:00:00+01:00</published>
  <updated>2018-02-20T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;This video started out as a practice run for a 3 minute lightning talk.&lt;/p&gt;

&lt;p&gt;I didn’t make it in 3 minutes.&lt;/p&gt;

&lt;p&gt;On the upside, there’s about 5 times more information in this talk than in the lightning talk it was a practice run for.&lt;/p&gt;

&lt;p class=&quot;videoWrapper&quot;&gt;
&lt;iframe width=&quot;680&quot; height=&quot;382&quot; src=&quot;https://www.youtube.com/embed/oJAZ7gBFf0U?cc_load_policy=1&amp;amp;cc_lang_pref=en&quot; frameborder=&quot;0&quot; allowfullscreen=&quot;&quot;&gt;&lt;/iframe&gt;
&lt;/p&gt;


  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/expressions-of-change-in-under-15-minutes/&quot;&gt;Expressions of Change in under 15 minutes&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on February 20, 2018.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Constructing S-Expressions]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/constructing-s-expressions/" />
  <id>http://www.expressionsofchange.org/constructing-s-expressions</id>
  <published>2017-12-11T00:00:00+01:00</published>
  <updated>2017-12-11T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;Putting the methods of construction more central does not mean we can ignore the programs that are constructed by them. For example: when we edit a program, the programmer is still presented with an actual program on the screen, when we evaluate a program, it is a single particular program that’s being evaluated rather than a history of programs etc.&lt;/p&gt;

&lt;p&gt;In short: an essential piece of any set of tools that takes changes to programs as its point of departure is to actually construct programs from those changes. When compared to toolchains which simply store already-constructed programs this is quite obviously an additional task. If only for that reason, it is worthwhile to consider its implementation and performance characteristics.&lt;/p&gt;

&lt;p&gt;In the below we present an algorithm to efficiently construct an s-expression out of a previous s-expression and a single modification. Before we turn our attention to that algorithm, we shall be a bit more precise about the properties of the data structure to be created.&lt;/p&gt;

&lt;h3 id=&quot;immutability-of-input&quot;&gt;Immutability of input&lt;/h3&gt;

&lt;p&gt;The presented algorithm is a pure function of an s-expression and a mutation to a new s-expression. In other words: the s-expression that serves as our input is not modified in-place. Any of its parts that are required in the output are copied as necessary. The usual advantages of functional programming thus apply, i.e the function is easier to test and reason about, thread-safe by default, and trivial to memoize.&lt;/p&gt;

&lt;p&gt;In the present project there is one more advantage to this approach: it provides for a trivial mechanism to construct a history of structures, one for each modification. Given the core assumptions of the present project, such a structure is extremely useful. To create it we simply keep a reference to each produced structure in a list. An algorithm that is formulated in terms of an in-place modification of a given input does not allow for this approach for the obvious reason that the referenced “historic” objects are not guaranteed to remain unchanged.&lt;/p&gt;

&lt;h3 id=&quot;data-structure&quot;&gt;Data structure&lt;/h3&gt;

&lt;p&gt;The list-expressions to be constructed shall be represented in-memory as an array of references to child nodes. This allows for lookup of a child by index in constant time.&lt;/p&gt;

&lt;p&gt;A number of alternatives to this design are easily rejected, as detailed below.&lt;/p&gt;

&lt;p&gt;An alternative to an array is formed by a linked list. However, representation of the list of children using a linked list has no advantages, only disadvantages. First, because of the choice for an immutable data structure, the potential advantages in performance for updates to the list do not apply. Second, lookup by index, the main method of access, is not in constant time for linked lists.&lt;/p&gt;

&lt;p&gt;An alternative to storing references to child nodes is formed by storing the child nodes inline in some serialized format. Such an approach may be useful for serialization on disk or over the wire, but is not sufficiently flexible in the face of modifications: each change to a child tree requires a full reserialization of the whole tree, with performance characteristics in the order of the size of the full tree.&lt;/p&gt;

&lt;h2 id=&quot;the-algorithm&quot;&gt;The algorithm&lt;/h2&gt;

&lt;p&gt;Before turning our attention to constructing s-expressions in terms of modifications to previous s-expressions, let us consider the properties of constructing s-expressions &lt;em&gt;per se&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Given an s-expression, construction of an s-expression in the data format outlined above can be formulated recursively: First, construct all child expressions with a recursive call; combine these results by constructing an array of references to them.&lt;/p&gt;

&lt;p&gt;This recursive definition can be expressed as a &lt;a href=&quot;/catamorphisms-and-change/&quot;&gt;catamorphism&lt;/a&gt;: the construction of children is independent of their parents. This catamorphism is in fact the most trivial catamorphism that exists: as a whole it is simply the identity function; its algebra is formed by what is basically a trivial data constructor.&lt;/p&gt;

&lt;p&gt;In &lt;a href=&quot;/catamorphisms-and-change/&quot;&gt;an earlier article&lt;/a&gt; we observed that in the context of controlled modification (i.e. a well-chosen &lt;em&gt;Clef&lt;/em&gt;) for recursive functions which can be described as catamorphisms efficient mechanisms for recalculation of modified versions are automatically available.&lt;/p&gt;

&lt;p&gt;Given that construction of s-expressions can be expressed as a catamorphism, construction of s-expressions in terms of modifications to previous s-expressions can be implemented using such automatically available efficient mechanisms with well-understood performance characteristics.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;/catamorphisms-and-change/#catamorphisms--controlled-modification&quot;&gt;The general mechanism&lt;/a&gt;, briefly summarized, is the following: for each modification, the calculation proceeds in two steps: first we calculate the outcome of the catamorphism for the small number of modified children and second we apply the algebra, i.e. we combine the previously calculated outcomes and newly calculated results.&lt;/p&gt;

&lt;p&gt;In this case, as noted, the algebra is simply the construction of an array of references to the children.&lt;/p&gt;

&lt;p&gt;The performance characteristics can be deduced from the performance characteristics of the algebra. In this case it is easy to see that the algebra is linear in the branching factor of the nodes: constructing an array of references to the children requires one action per child.&lt;/p&gt;

&lt;p&gt;As detailed in the article on catamorphisms and change, the total cost of applying a root-level modification is then the product of [a] the branching factor, [b] the depth of the tree and [c] the number of modifications at the deepest level.&lt;/p&gt;

&lt;p&gt;The run-time performance is bound by the time spent constructing the result; as a consequence, it is easy to see that the required storage space for a full history of all s-expressions is given by the same formula.&lt;/p&gt;

&lt;h3 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h3&gt;

&lt;p&gt;In &lt;em&gt;Expressions of Change&lt;/em&gt; mechanisms of construction are put center stage. The first and most practical application of such methods is the actual construction of s-expressions.&lt;/p&gt;

&lt;p&gt;In the above we presented an algorithm to do so; based on the observation that construction of s-expressions can be formulated as a trivial catamorphism.&lt;/p&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/constructing-s-expressions/&quot;&gt;Constructing S-Expressions&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on December 11, 2017.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Catamorphisms and change]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/catamorphisms-and-change/" />
  <id>http://www.expressionsofchange.org/catamorphisms-and-change</id>
  <published>2017-12-04T00:00:00+01:00</published>
  <updated>2017-12-04T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;Let us first establish an informal notion of a catamorphism.&lt;/p&gt;

&lt;p&gt;Some recursive algorithms on hierarchical data structures (i.e. trees) have the property that they are expressed as a function which [1] does a single recursive function call on each of its children, and [2] that this call takes no parameters other than the child node.&lt;/p&gt;

&lt;p&gt;The below is an example of such an algorithm in Python. It takes as its input a tree, which contains a number in the attribute &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;value&lt;/code&gt; at each node. The algorithm calculates the sum of these numbers:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;sum_of_values&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;value&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;child&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;children&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;sum_of_values&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;child&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Not every recursive algorithm on a data-structure can be so written. As an example consider a mechanism for pretty-printing an Abstract Syntax Tree, in which the first child of any node is always printed on a single line, and all other nodes are each printed on their own line.&lt;/p&gt;

&lt;p&gt;&lt;a name=&quot;pretty_print&quot;&gt;&lt;/a&gt;This function is also provided in sketched form in Python (we leave out some details):&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;pretty_print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;single_line_mode&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;False&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;single_line_mode&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt; &lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;join&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;([&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;pretty_print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;child&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;child&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;children&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;

    &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;children&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;pretty_print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;children&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;child&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;children&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:]:&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;pretty_print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;child&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;False&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;sh&quot;&gt;&quot;&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The key point of the example above is: The recursive call contains more information than simply the fact that some calculation must be done recursively. This information is not knowable from the child node, it is a property of its context. In this case: whether the present node must be printed on a single line as a result of being part of another node’s first element or not.&lt;/p&gt;

&lt;p&gt;Note that this is not just a problem with the chosen implementation, but a fundamental property of the desired output. In other words: we cannot rewrite the second example to be in the shape of the first example. In particular: rewriting the above by replacing the boolean parameter with 2 separate function calls is not a valid solution, because then the algorithm is no longer expressed in terms of a single recursive function call on each of the children.&lt;/p&gt;

&lt;p&gt;So we can see a distinction: algorithms in which we can do calculations for constituent parts of a composite data structure independently from the data structure of which they are a part, and those for which this is not possible.&lt;/p&gt;

&lt;p&gt;Recursive algorithms of the first kind are called &lt;a href=&quot;https://en.wikipedia.org/wiki/Catamorphism&quot;&gt;catamorphisms&lt;/a&gt;.&lt;sup id=&quot;fnref:more-about-catamorphisms&quot;&gt;&lt;a href=&quot;#fn:more-about-catamorphisms&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Catamorphisms are often described as “generalized folding over arbitrary data structures”. The mechanism of folding the results from the children into a single result is called the &lt;em&gt;algebra&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;The function that describes this algebra can be factored out from the catamorphism. The function describing the algebra is quite similar to the original function, but the node being passed to it has as its children not further nodes, but the results of recursive application.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;sum_of_values_algebra&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;value&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;child_result&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;children&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;child_result&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt;


&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;catamorphism&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;algebra&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;child_results&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[]&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;child&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;children&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;child_results&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;append&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;catamorphism&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;algebra&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;child&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;algebra&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nc&quot;&gt;Node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;value&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;child_results&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;


&lt;span class=&quot;n&quot;&gt;sum_of_values&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;lambda&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;catamorphism&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sum_of_values_algebra&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;This separates the concerns of recursively calling (which is in the utility function &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;node_catamorphism&lt;/code&gt;) and composing the results (which is in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;sum_of_values_algebra&lt;/code&gt;).&lt;/p&gt;

&lt;h3 id=&quot;working-examples&quot;&gt;Working examples&lt;/h3&gt;

&lt;p&gt;The &lt;a href=&quot;/introducing-the-editor/&quot;&gt;editor that is part of this project&lt;/a&gt; is a structured editor of s-expressions. In the below some catamorphisms that can be defined for an s-expression are given as working examples. Determining that these are indeed catamorphisms is left as an exercise to the reader.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Calculating the size of an s-expression; either defined as the number of nodes it contains or the total in-memory size needed for storage.&lt;/li&gt;
  &lt;li&gt;Serialization of an s-expression for storage on-disk or in-memory.&lt;/li&gt;
  &lt;li&gt;Counting occurrences of specific terms, e.g. “total number of occurrences of the symbol &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;lambda&lt;/code&gt; in this s-expression”.&lt;/li&gt;
  &lt;li&gt;Calculating the set of symbols occurring in the s-expression.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;As should be obvious from the introduction, not every algorithm on an s-expression can be rewritten to a catamorphism. Here are some examples:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Pretty-printing; at least not in the general case, in which the way a node is printed might depend on its ancestors.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;/form-analysis/&quot;&gt;Form-analysis&lt;/a&gt;; whether any given s-expression must be analyzed as a Lisp-form &lt;em&gt;at all&lt;/em&gt; depends on ancestors.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;context-controlled-modification&quot;&gt;Context: controlled modification&lt;/h3&gt;

&lt;p&gt;In the project that this website is about, modifications to programs are believed to be such a central aspect of the programming experience that they are given center stage in the tooling. This is most clearly the case for &lt;a href=&quot;/introducing-the-editor/&quot;&gt;the editor&lt;/a&gt;, which has as its primary output not a document (in our case: an s-expression), but rather a well-structured history describing how to create such a document in by steps, each step representing a single modification.&lt;/p&gt;

&lt;p&gt;The possible kinds of different modifications are described in a &lt;a href=&quot;/introducing-the-editor/#a-clef-for-construction&quot;&gt;Clef&lt;/a&gt;, a “vocabulary of change”. We now make an important observation about this Clef:&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Each single modification to a list-expression only affects a very limited number of that list’s elements&lt;/em&gt;;&lt;/p&gt;

&lt;p&gt;In fact, in the Clef as presented so far each modification only affects at most a single child, in one of three ways:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;A single existing child is deleted.&lt;/li&gt;
  &lt;li&gt;A single existing child is replaced.&lt;/li&gt;
  &lt;li&gt;A single new child is inserted.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It is certainly possible that the Clef will be extended at some point in the future, for example to include a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Swap&lt;/code&gt; operation. However, the general property, that notes from the &lt;em&gt;clef&lt;/em&gt; act on a specific subset of children of the s-expression under consideration, while not changing any of the other children, will always be preserved: it is a key part of the design.&lt;/p&gt;

&lt;h3 id=&quot;catamorphisms--controlled-modification&quot;&gt;Catamorphisms &amp;amp; controlled modification&lt;/h3&gt;

&lt;p&gt;In the context of controlled modification new outcomes for catamorphisms on a structure can be calculated quite efficiently. To see how this is so, consider the following.&lt;/p&gt;

&lt;p&gt;If the algorithm can be expressed as a catamorphism, the calculation that needs to be done for a particular child node depends on no other information than the information contained in the child node – this is so by the definition of catamorphisms.&lt;/p&gt;

&lt;p&gt;Each modification affects only a very limited number of children – as a consequence of a carefully chosen &lt;em&gt;Clef&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;The other children are unaffected, which means that the outcome of the catamorphism will also be unaffected for them.&lt;/p&gt;

&lt;p&gt;This means that, for any modification, we only need to recalculate [1] the outcome of the catamorphism for the small number of affected children and [2] the combination of previously calculated outcomes and new results at the modified node itself, i.e. the application of the algebra.&lt;/p&gt;

&lt;p&gt;To see this in action, consider the catamorphism of “total size” which is depicted graphically below. In this picture each node is annotated a size showing the total size of that subtree:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/node-size.jpg&quot; alt=&quot;Tree with &amp;quot;total size&amp;quot;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Replacing the right-most child of the root node is depicted below. A red arrow with an “R” depicts the replacing of one node by another; Green arrows depict the fact that recalculation at the root node considers the values at that node’s direct children; a &lt;em&gt;9&lt;/em&gt; at the root represents the updated outcome of the calculation.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/node-size-update.jpg&quot; alt=&quot;Tree with &amp;quot;total size&amp;quot; - update&quot; /&gt;&lt;/p&gt;

&lt;p&gt;The fact that no green arrows point at the leaf-nodes on the bottom left represents the efficiency-gain with respect to a full recalculation of the catamorphism on the new input structure.&lt;/p&gt;

&lt;p&gt;In this example, the effect is not all too dramatic, because the tree is relatively small; in the general case, dramatic efficiency gains will be made precisely when they are required, namely when the trees are very large.&lt;/p&gt;

&lt;h2 id=&quot;performance-characteristics&quot;&gt;Performance characteristics&lt;/h2&gt;

&lt;p&gt;Let us explore the computational cost of a single modification. This cost is built up from two separate parts:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Recursively constructing a new child.&lt;/li&gt;
  &lt;li&gt;Combining the results of the children.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;recursive-construction&quot;&gt;Recursive construction&lt;/h3&gt;

&lt;p&gt;We start by considering the cost of recursively computing the result for a new child.&lt;/p&gt;

&lt;p&gt;Note that for &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Delete&lt;/code&gt; there is no new child, and hence there is no cost. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; both take as one of their parameters a history: a sequence of modifications.&lt;/p&gt;

&lt;p&gt;The cost of recursive construction of a child is then: the sum of the costs for each modification in this history. That is to say: the cost of the step-wise recursive application of the algorithm. The cost of a each modification in the history is thus recursively described by the section “Performance characteristics” of this article.&lt;/p&gt;

&lt;p&gt;The other factor is the length of the history for which recalculations must be done.&lt;/p&gt;

&lt;p&gt;In the case of a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; it is the number of &lt;em&gt;new&lt;/em&gt; items in the provided history at least if we apply the constraint that replace can only replace with &lt;a href=&quot;/constraints-on-replace/&quot;&gt;a continuation of the existing history&lt;/a&gt;. That is because only for new items in the provided history recalculations need to be done.&lt;/p&gt;

&lt;p&gt;In the case of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert&lt;/code&gt; this is the full length of the provided history: no earlier result is available, so we must calculate from scratch.&lt;/p&gt;

&lt;h3 id=&quot;combining-results-the-algebra&quot;&gt;Combining results: the algebra&lt;/h3&gt;

&lt;p&gt;The computational cost of each call to the &lt;em&gt;algebra&lt;/em&gt; depends on the algebra at hand.&lt;/p&gt;

&lt;p&gt;Let’s consider two examples with rather different computational complexity to see how this is the case:&lt;/p&gt;

&lt;p&gt;A first example is the summation of values shown at the beginning of this article. It scales linearly with the number of children (branching factor). This is because while taking the sum over all children we must consider each child at least once.&lt;/p&gt;

&lt;p&gt;Contrast this with the serialization of a data structure into a single piece of contiguous memory. That scales linearly with the sum of the child result lengths (which is the same as to say: scales linearly with the output’s length). This is because each serialized child needs to be fully included in the result. Copying each child takes a time linear with the child’s size.&lt;/p&gt;

&lt;p&gt;Now let’s consider how the computational cost of the algebra affects the computational cost of the algorithm as a whole. For the algorithm of selective recalculation of a catamorphic function outlined above, the computational cost of the algebra is recursively incurred at each level of the tree. For algebras that have a computational cost that is linear with the branching factor, such as the sum-of-values, this means that the total cost is in the order of the product of the following 3 things:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;The average branching factor of the tree&lt;/li&gt;
  &lt;li&gt;The depth of the tree (more precisely: the deepest level where modification takes place)&lt;/li&gt;
  &lt;li&gt;The number of modifications at the deepest level.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For large trees with small modifications this implies a great efficiency gain in comparison to a full recalculation.&lt;/p&gt;

&lt;p&gt;Note that the above total cost of the algorithm is only applicable when the cost of the algebra is linear with the branching factor.&lt;/p&gt;

&lt;p&gt;Serialization into contiguous memory, as detailed above, is an important example of this condition not holding. As a result, it is better not to implement it using selective recalculation. Recalculation from scratch at each step is significantly cheaper.&lt;/p&gt;

&lt;h3 id=&quot;incremental-calculation-of-algebras&quot;&gt;Incremental calculation of algebras&lt;/h3&gt;

&lt;p&gt;In the above approach, the performance gains with respect to a full recalculation are made by selectively recursing: results that are already known are not recalculated.&lt;/p&gt;

&lt;p&gt;However, the result of the algebraic function is calculated precisely as in the non-incremental case: at each node at which recalculation takes place we do a full recomputation of the algebraic function with all its inputs.&lt;/p&gt;

&lt;p&gt;For each such recomputation, the branching factor forms a lower bound on the performance: the inputs must at least be collected and passed to the algebra, and each child forms an input. For the algorithm as a whole, this lower bound is reflected as one of the 3 factors mentioned in the bulleted list above.&lt;/p&gt;

&lt;p&gt;This raises the question: is it possible to arrive at the result of the algebraic function without considering all its inputs?&lt;/p&gt;

&lt;p&gt;Let us revisit the example of total tree-size, as mentioned above, to see how the answer to that question may be “yes”.&lt;/p&gt;

&lt;p&gt;Consider again the graphical example in the above. In it, a node with a total size of &lt;em&gt;1&lt;/em&gt; is replaced with a node with a value of &lt;em&gt;2&lt;/em&gt;. From this fact alone, we can conclude that the total tree size is increased by &lt;em&gt;1&lt;/em&gt; (namely: &lt;em&gt;2 - 1&lt;/em&gt;). Using the knowledge that the previous tree size was &lt;em&gt;8&lt;/em&gt;, we can conclude that the new tree size will be &lt;em&gt;9&lt;/em&gt;. This conclusion can be reached without referring to the two existing nodes with value &lt;em&gt;3&lt;/em&gt;. In other words: the branching factor does not come into play.&lt;/p&gt;

&lt;p&gt;In general, we may sometimes have access to a mechanism of recalculation of the algebraic function that is not bound by the branching factor. This is made possible by taking a previous result of the algebraic function, as well as structred updates (deletions, insertions, replacements), into account. In other words: such mechanisms take an incremnental approach to the algebraic function.&lt;/p&gt;

&lt;p&gt;In general, such a mechanism will always be available for a given catamorphism given two conditions:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;The outcome of combining results is unaffected the order of the children. (An example of this condition not holding: propagation of the left-most symbol occurring in a tree to its root)&lt;/li&gt;
  &lt;li&gt;The effect of removing a single child can be expressed without taking into account the other children. (An example of this condition not holding: calculating the &lt;em&gt;set&lt;/em&gt; of unique symbols)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h2&gt;

&lt;p&gt;In this article we have taken a look at catamorphisms in the context of controlled modification.&lt;/p&gt;

&lt;p&gt;Some recursive functions can be expressed as a catamorphism. If the computational complexity of the associated algebra scales sub-linearly with the size of the underlying tree, an algorithm for efficient stepwise recalculation is available.&lt;/p&gt;

&lt;p&gt;For some algebras we can do even better: we can ignore the values of the presently existing children entirely, and calculate a new value based purely by applying the results of deletion and insertion.&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:more-about-catamorphisms&quot;&gt;
      &lt;p&gt;The canonical introduction is &lt;a href=&quot;https://research.utwente.nl/en/publications/functional-programming-with-bananas-lenses-envelopes-and-barbed-w&quot;&gt;Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire&lt;/a&gt;, but the squiggol notation may make it somewhat hard to understand for readers not familiar with that notation. The article by &lt;a href=&quot;https://bartoszmilewski.com/2013/06/10/understanding-f-algebras/&quot;&gt;Bartosz Milewski&lt;/a&gt; is also excellent. &lt;a href=&quot;#fnref:more-about-catamorphisms&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/catamorphisms-and-change/&quot;&gt;Catamorphisms and change&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on December 04, 2017.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Static Form Analysis]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/form-analysis/" />
  <id>http://www.expressionsofchange.org/form-analysis</id>
  <published>2017-11-17T00:00:00+01:00</published>
  <updated>2017-11-17T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;To understand what we mean by form-analysis, and why it could be useful, consider again the &lt;a href=&quot;/l-a-toy-language/#case-analysis&quot;&gt;model for evaluation presented in the previous article&lt;/a&gt;, which provides a definition of evaluation of s-expressions in environments.&lt;/p&gt;

&lt;p&gt;A first step in this evaluation is a case-analysis on the form of the s-expression. For each evaluated s-expression, we first check whether the s-expression is an if-statement, lambda, function application or any of the other forms; only then we proceed accordingly. This case-analysis is reproduced below for reference, leaving out the semantics of evaluation for reasons of brevity.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;an atom which is either a string or a numbers is a &lt;strong&gt;value&lt;/strong&gt;&lt;/li&gt;
  &lt;li&gt;any other atom is a &lt;strong&gt;variable&lt;/strong&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(quote «s-expr»)&lt;/code&gt; is a &lt;strong&gt;quoted expression&lt;/strong&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(quote «predicate» «consequent» «alternative»)&lt;/code&gt; is an &lt;strong&gt;if-expression&lt;/strong&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(define «variable-name» «definition»)&lt;/code&gt; is a &lt;strong&gt;definition&lt;/strong&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(lambda «parameter names» «body ...»)&lt;/code&gt; is a &lt;strong&gt;lambda&lt;/strong&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(begin «... ...»)&lt;/code&gt; is a &lt;strong&gt;sequence&lt;/strong&gt;&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(«procedure» «arguments ...»)&lt;/code&gt; is a &lt;strong&gt;procedure application&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In the process of evaluation of s-expressions, any given s-expression may be evaluated multiple times. Consider for example how, in &lt;a href=&quot;/l-a-toy-language/#an-example&quot;&gt;our running example&lt;/a&gt; of the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;factorial&lt;/code&gt; procedure, when called with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(factorial 5)&lt;/code&gt;, the body of that procedure is subjected to the case analysis for each invocation, for a total of 5 times. An interpreter that implements the above definition of evaluation directly, i.e. by operating on s-expressions, will repeat the case-analysis for each such evaluation.&lt;/p&gt;

&lt;h3 id=&quot;static-nature-of-the-problem&quot;&gt;Static nature of the problem&lt;/h3&gt;

&lt;p&gt;However, the outcome of the analysis depends exclusively on certain static, structural aspects of the s-expression under consideration: first, whether it is an atom or a list; second, in case it’s a list, whether its first element is one of a specific set of atoms (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;lambda&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;if&lt;/code&gt;, …).&lt;/p&gt;

&lt;p&gt;This means that, for any given s-expression, the result of the case-analysis will always be the same, independent of any dynamic aspects of the evaluation. Another way to say this is: the case-analysis can be done &lt;em&gt;statically&lt;/em&gt;.&lt;sup id=&quot;fnref:sicp-form-analysis&quot;&gt;&lt;a href=&quot;#fn:sicp-form-analysis&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; That is, given an s-expression, we can identify which form it has, and recursively which forms that form is composed of.  The output of such an analysis can be used as the input for a mechanism of evaluation.&lt;/p&gt;

&lt;h3 id=&quot;why-do-static-analysis&quot;&gt;Why do static analysis?&lt;/h3&gt;

&lt;p&gt;Doing this is useful for a number of reasons:&lt;/p&gt;

&lt;p&gt;First, as implied in the above, an interpreter which statically analyses its s-expression only once, before evaluation, can avoid repeating the same case analysis over an over at evaluation-time. This is beneficial for the interpreter’s performance.&lt;/p&gt;

&lt;p&gt;Second, not every form is a legal form. For example, an s-expression denoting an &lt;em&gt;if-form&lt;/em&gt; must have precisely 4 elements: [1] the symbol &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;if&lt;/code&gt;, [2] a &lt;em&gt;predicate&lt;/em&gt;, [3] a &lt;em&gt;consequent&lt;/em&gt; and [4] an &lt;em&gt;alternative&lt;/em&gt;; any other number of elements is illegal. A static analysis of forms will bring any malformed forms to light before evaluation-time. This means that such errors can be corrected sooner and with lower cost.&lt;/p&gt;

&lt;p&gt;Third, a static model of the program’s forms is a prerequisite for nearly all other forms of static analysis. We better do it before proceeding.&lt;/p&gt;

&lt;p&gt;A final argument for a separate phase of static form analysis can be found in the understanding of the material: doing form-analysis statically is trivial enough to be a good introduction into the subject of static analysis, but still non-trivial enough to provide some insights that can be useful when considering later analyses.&lt;/p&gt;

&lt;h3 id=&quot;static-analysis-a-sketch&quot;&gt;Static analysis: a sketch&lt;/h3&gt;

&lt;p&gt;In the below we provide a rough sketch of a static form analysis.&lt;/p&gt;

&lt;p&gt;Static form analysis is defined recursively on s-expressions. For each s-expression under consideration we do a case-analysis on the structure of the s-expression. The patterns on which we match correspond precisely to those enumerated above, i.e. we distinguish between various kinds of atoms, and for list-expressions we make a distinction based on their first element (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;lambda&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;if&lt;/code&gt;, …).&lt;/p&gt;

&lt;p&gt;For each matched pattern an object of a specific type, with attributes which are specific to the form at hand, is constructed. For example, for an &lt;em&gt;if-form&lt;/em&gt; we construct an &lt;em&gt;if-form&lt;/em&gt; object, which has the 3 attributes &lt;em&gt;predicte&lt;/em&gt;, &lt;em&gt;consequent&lt;/em&gt; and &lt;em&gt;alternative&lt;/em&gt;, each themselves being a form object.&lt;sup id=&quot;fnref:object&quot;&gt;&lt;a href=&quot;#fn:object&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;To construct the sub-forms out of s-expressions, we recurse. For most cases, the paths of recursion are analogous to those for evaluation. Some examples:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;for an &lt;em&gt;if-form&lt;/em&gt; we recursively analyze the &lt;em&gt;predicate&lt;/em&gt;, &lt;em&gt;consequent&lt;/em&gt; and &lt;em&gt;alternative&lt;/em&gt;.&lt;/li&gt;
  &lt;li&gt;for a &lt;em&gt;sequence&lt;/em&gt; we recursively analyze all s-expressions that make up the sequence.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For two cases, however, the recursion follows a different path than that of evaluation.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;For &lt;em&gt;lambda-form&lt;/em&gt; we recursively analyze the body of the form – contrast this with the case of evaluation, in which the body of a lambda is not evaluated when the lambda is evaluated but instead stored in a procedure object.&lt;/li&gt;
  &lt;li&gt;Analysis of function-application recursively analyses the application’s sub-expressions – contrast this with the case of evaluation, which proceeds to evaluate the body of the procedure object.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These two differences from the recursive definition of evaluation have the important consequence that the analysis is linear in time with the size of the s-expression under consideration including all its descendants – contrast this with evaluation, which may take infinite time.&lt;/p&gt;

&lt;p&gt;Finally, a word on the well-formedness of forms: for forms which are not well-formed, such as a list-expression representing an &lt;em&gt;if-form&lt;/em&gt; with any other number than 4 elements, a special &lt;em&gt;malformed-form&lt;/em&gt; is constructed. For malformed forms we do not recurse.&lt;/p&gt;

&lt;h3 id=&quot;context-dependent-analysis&quot;&gt;Context-dependent analysis&lt;/h3&gt;

&lt;p&gt;Nota bene: the fact that the analysis is recursively defined does not imply that an analysis of a given list-expression always recurses into all elements of that list. Instead, the paths of recursion are dependent on the form currently under consideration. In particular, the following sub-expressions are not subjected to further form-analysis at all:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;the form-identifying atoms (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;if&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;lambda&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;begin&lt;/code&gt; etc.) – these atoms determine the form associated with the s-expression containing them, but are not themselves subjected to further analysis.&lt;/li&gt;
  &lt;li&gt;the quoted s-expressions of &lt;em&gt;quote&lt;/em&gt; forms – the point of quoting is precisely that the quoted s-expression is not to be seen as a form: it is stored as-is as part of the quoted form.&lt;/li&gt;
  &lt;li&gt;the single variable name in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;define&lt;/code&gt; – this variable name is stored as-is as an attribute of the definition.&lt;/li&gt;
  &lt;li&gt;the list of parameter-names in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;lambda&lt;/code&gt; – stored as-is as a list of names.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This implies that we &lt;em&gt;must&lt;/em&gt; approach the analysis in a top-down manner, because even the question as to whether we must analyze an s-expression to begin with is entirely dependent on its location in the tree.&lt;/p&gt;

&lt;h3 id=&quot;as-a-picture&quot;&gt;As a picture&lt;/h3&gt;

&lt;p&gt;Readers who prefer pictures over words may find the following section useful. In it, we show two diagrams.&lt;/p&gt;

&lt;p&gt;In the first diagram an s-expression is presented, representing the expression &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(if (= 1 2) 3 4)&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;In this diagram list-expressions simply show up as pairs of brackets, i.e. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;()&lt;/code&gt;, with the elements of those expressions below them.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/if-s-expr.jpg&quot; alt=&quot;An s-expression representing an if&quot; /&gt;&lt;/p&gt;

&lt;p&gt;The second diagram represents the output of the form-analysis. In it, each special form has been recognized and labeled as such. The outgoing branches of the tree also have labels, which denote the specific role of each child-node in the tree.&lt;/p&gt;

&lt;p&gt;Note that there is no 1-to-1 correspondence of nodes amongst the two trees. For example: the top-level s-expression &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;()&lt;/code&gt; and its first element, the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;if&lt;/code&gt;, are combined into a single node in the tree representing form-analysis. Similarly: both the atoms &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;1&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;2&lt;/code&gt; are folded into a single attribute of the function application representing all the function’s arguments.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/if-form.jpg&quot; alt=&quot;The associated form&quot; /&gt;&lt;/p&gt;

&lt;h3 id=&quot;terminology&quot;&gt;Terminology&lt;/h3&gt;

&lt;p&gt;Finally, a few words on the chosen terminology.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://mitpress.mit.edu/sites/default/files/6515.pdf&quot;&gt;Sussman &amp;amp; Abelson&lt;/a&gt; call the phase described here &lt;em&gt;syntactic analysis&lt;/em&gt;. We have not followed that tradition, mainly because it is not obvious what noun to use to refer to the analyzed objects in that case.&lt;/p&gt;

&lt;p&gt;Although the present phase has some properties that a reminiscent of “parsing”, we have refrained from using that word here, because it is generally associated with the parsing of strings – a phase which is entirely absent in the present project, given the fact that our point of departure is already a structured editor of s-expressions.&lt;/p&gt;

&lt;p&gt;“Abstract Syntax Tree” was also ruled out as a term, because it does not tell us whether it’s the abstract syntax of s-expressions (the input of the current step) or the abstract syntax of forms (the output of the current step).&lt;/p&gt;

&lt;p&gt;Regarding the term “&lt;em&gt;forms&lt;/em&gt;” the following: In Lisp Forms are &lt;a href=&quot;http://www.gnu.org/software/emacs/manual/html_node/elisp/Intro-Eval.html&quot;&gt;“any Lisp object that is intended to be evaluated”&lt;/a&gt;. Here we additionally mean “containing the relevant information on how this evaluation may take place given the object’s position in the full program tree”. Some people mean “Lisp Form” to be an implication of static correctness of the form. We don’t do that, and instead say that incorrect forms are a subset of forms. They are simply those lisp objects that are intended to be evaluated, but malformed.&lt;/p&gt;

&lt;h3 id=&quot;concluding-remarks&quot;&gt;Concluding remarks&lt;/h3&gt;

&lt;p&gt;In this article, we have seen why static form analysis could be useful and how it could be implemented.&lt;/p&gt;

&lt;p&gt;So far, we have only talked about the form-analysis of any given s-expression.&lt;/p&gt;

&lt;p&gt;What we can want next is to implement an analysis that does a minimal amount of re-analyzing, given a previous version of the program, a completed analysis and a small change to the program.&lt;/p&gt;

&lt;p&gt;Such an incremental form analysis is the subject of the next (as of yet to be written) article.&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:sicp-form-analysis&quot;&gt;
      &lt;p&gt;This is not a unique idea; it is, for example, explored in quite some detail in the &lt;a href=&quot;https://mitpress.mit.edu/sites/default/files/6515.pdf&quot;&gt;Structure and Interpretation of Computer Programs&lt;/a&gt; in chapter 4.1.7, “Separating Syntactic Analysis from Execution”, p. 328. &lt;a href=&quot;#fnref:sicp-form-analysis&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:object&quot;&gt;
      &lt;p&gt;How such an object is implemented depends on the language of implementation, e.g. in strongly typed functional languages one may implement such objects using algebraic data types, in Lisps using closures etc. &lt;a href=&quot;#fnref:object&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/form-analysis/&quot;&gt;Static Form Analysis&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on November 17, 2017.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[L - A toy language]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/l-a-toy-language/" />
  <id>http://www.expressionsofchange.org/l-a-toy-language</id>
  <published>2017-11-16T00:00:00+01:00</published>
  <updated>2017-11-16T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;The &lt;a href=&quot;/the-editor/&quot;&gt;editor that is part of this project&lt;/a&gt; puts changes to documents more centrally in the editing experience than traditional editors. In defining a language for static analysis, however, we shall start by focusing on the structure that can be constructed from these changes: an s-expression.&lt;/p&gt;

&lt;p&gt;S-expressions are perhaps most well-known for being the basic building block of the syntax of Lisp and its various dialects. In this series we will stay true to that tradition, defining a minimal Lisp as the basis for further discussion. In the below, we shall refer to this language as &lt;em&gt;L&lt;/em&gt;. Despite being quite minimal, &lt;em&gt;L&lt;/em&gt; is sufficiently complete for actual usage, and for various static analyses to yield interesting results.&lt;/p&gt;

&lt;p&gt;The given language is a subset of Scheme. In the interest of brevity the definition below is somewhat imprecise; for a full definition of Scheme the reader is referred to &lt;a href=&quot;https://mitpress.mit.edu/sites/default/files/6515.pdf&quot;&gt;the excellent book by Abelson &amp;amp; Sussman&lt;/a&gt; or the accompanying &lt;a href=&quot;https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-001-structure-and-interpretation-of-computer-programs-spring-2005/video-lectures/&quot;&gt;video lectures&lt;/a&gt;. A more precise definition of the language under consideration here may be obtained by taking Scheme as defined in those sources, and limiting it to the special forms as defined below.&lt;/p&gt;

&lt;h2 id=&quot;operational-semantics&quot;&gt;Operational semantics&lt;/h2&gt;

&lt;p&gt;In the below we provide an &lt;em&gt;operational semantics&lt;/em&gt;, which is to say: the meaning of the language is entirely defined by its behavior in some imaginary machine. In particular, given an s-expression and an environment, we provide a mechanism for evaluation.&lt;/p&gt;

&lt;p&gt;S-expressions have been talked about at length &lt;a href=&quot;/introducing-the-editor/&quot;&gt;in our description of the editor&lt;/a&gt;; With regards to environments, it suffices to say two things:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Environments bind values to names; after a value has been bound to a name, looking up the name returns the value.&lt;/li&gt;
  &lt;li&gt;Environments may be nested; if a name is looked up in a nested environment and not found there, lookup recursively proceeds in the enclosing environment.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;case-analysis&quot;&gt;Case-analysis&lt;/h3&gt;

&lt;p&gt;We define evaluation in terms of a case-analysis on the shape of the s-expression. For list expressions, we use a pattern-matching-like notation for this shape; the reader is expected to be able to either deal with the lack of precision, or look up a more precise definition in the above-mentioned sources. Note that the definition is recursive: some of the mechanisms of evaluation below are defined in terms of further evaluation.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;an atom which is either a string or a numbers is a &lt;strong&gt;value&lt;/strong&gt;; its evaluation results in this value itself.&lt;/li&gt;
  &lt;li&gt;any other atom is a &lt;strong&gt;variable&lt;/strong&gt;; its evaluation results in the value resulting from looking up this variable in the present environment.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(quote «s-expr»)&lt;/code&gt; is a &lt;strong&gt;quoted expression&lt;/strong&gt;; its evaluation results in the quoted s-expression as-is (i.e., specifically by not further evaluation the quoted s-expression).&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(quote «predicate» «consequent» «alternative»)&lt;/code&gt; is an &lt;strong&gt;if-expression&lt;/strong&gt;; its evaluation depends on the value resulting from evaluating the &lt;em&gt;predicate&lt;/em&gt;; if this is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;true&lt;/code&gt;, it is the result of evaluating of the &lt;em&gt;conseqeuent&lt;/em&gt;, otherwise that of evaluating the &lt;em&gt;alternative&lt;/em&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(define «variable-name» «definition»)&lt;/code&gt; is a &lt;strong&gt;definition&lt;/strong&gt;; its evaluation does not result in a (meaningful) value; however, as an effect of evaluating the result of evaluating the definition is bound to the variable-name in the present environment.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(lambda «parameter names» «body ...»)&lt;/code&gt; is a &lt;strong&gt;lambda&lt;/strong&gt;; it evaluates to a procedure object which contains the parameter names and body. (Note: unlike in Scheme, in which a special form of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;define&lt;/code&gt; is interpreted as a procedure, in &lt;em&gt;L&lt;/em&gt; lambda’s can only be defined directly using the associated special form)&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(begin «... ...»)&lt;/code&gt; is a &lt;strong&gt;sequence&lt;/strong&gt;; it evaluates each s-expression in turn, the evaluation of the sequence results in the last element’s evaluation.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(«procedure» «arguments ...»)&lt;/code&gt; is a &lt;strong&gt;procedure application&lt;/strong&gt;; evaluation results in the value that is arrived at as follows: first, evaluate the procedure and all arguments in the present environment. Create a new (sub-)environment in which each of the procedure’s parameter names is bound to the value of the associated argument. Evaluate the procedure’s body in this environment. This is the result of evaluating the procedure application.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For the sake of simplicity, &lt;em&gt;L&lt;/em&gt; has no primitive procedures; this is no problem, since we can always evaluate programs in some “global” environment in which all required primitive procedures are bound to some name.&lt;/p&gt;

&lt;h3 id=&quot;an-example&quot;&gt;An example&lt;/h3&gt;

&lt;p&gt;As an example, given this definition, consider the following s-expression:&lt;/p&gt;

&lt;div class=&quot;language-scheme highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;begin&lt;/span&gt;
  &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;define&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;factorial&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;lambda&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;n&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
      &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;n&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;n&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;factorial&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;n&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))))))&lt;/span&gt;
  &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;factorial&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Evaluating this in an environment where the mathematical symbols are bound to procedures expressing their usual meaning yields the value &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;120&lt;/code&gt;.&lt;/p&gt;

&lt;h3 id=&quot;conclusions&quot;&gt;Conclusions&lt;/h3&gt;

&lt;p&gt;This concludes the definition of the language &lt;em&gt;L&lt;/em&gt;. In the next &lt;a href=&quot;/form-analysis/&quot;&gt;article in this series&lt;/a&gt; we will see a very first example of a static analysis on programs in &lt;em&gt;L&lt;/em&gt;, namely the analysis on the forms in such programs.&lt;/p&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/l-a-toy-language/&quot;&gt;L - A toy language&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on November 16, 2017.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Static Analysis - Introduction]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/static-analysis-intro/" />
  <id>http://www.expressionsofchange.org/static-analysis-intro</id>
  <published>2017-11-15T00:00:00+01:00</published>
  <updated>2017-11-15T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;Static program analysis is the analysis of computer programs that is performed without actually executing programs. Some examples are:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;identifying variables that are declared, but never used;&lt;/li&gt;
  &lt;li&gt;identifying variables that are used, but never declared;&lt;/li&gt;
  &lt;li&gt;any kind of type-checker.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Such analyses are useful in helping the programmer identify potential errors as soon as possible.&lt;/p&gt;

&lt;p&gt;Case in point: running a program with undeclared variables will lead to failure at the point the variable is used, which is why running such programs is ruled out in many (but not all) programming environments.&lt;/p&gt;

&lt;p&gt;Let’s see how such analyses relate to the present project. This project is about &lt;em&gt;Expressions of Change&lt;/em&gt;: environments in which the changes to programs take a central role, e.g. by having an editor which stores histories of structured changes to documents rather than the resulting documents.&lt;/p&gt;

&lt;p&gt;In the context of such an editor, doing such static analyses for each version of the program yields an extra dimension of valuable information: in addition to giving information about &lt;em&gt;what&lt;/em&gt; is wrong and &lt;em&gt;where&lt;/em&gt;, such step by step analyses provide insight into &lt;em&gt;when&lt;/em&gt; something went wrong, in terms of the edit-history of your program. This extra dimension may be quite helpful in answering the most important question, which is the question as to &lt;em&gt;why&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;As an example, consider an analysis which identifies (attempted) usage of undeclared variables. A traditional such analysis will point at the location in our program where an unknown variable is used. Doing the same analysis for each version of the program (where &lt;em&gt;every&lt;/em&gt; modification of the program yields a new version) will help us pinpoint &lt;em&gt;when&lt;/em&gt; the problem was introduced. In other words: it will pinpoint a precise modification of the program. By examining this modification, the question of &lt;em&gt;why&lt;/em&gt; may be more readily answered. Examples of possible answers are:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;the variable was renamed at its usage-location;&lt;/li&gt;
  &lt;li&gt;the variable was renamed at the location of its definition;&lt;/li&gt;
  &lt;li&gt;the location of the variable’s definition was removed entirely;&lt;/li&gt;
  &lt;li&gt;the variable has never been defined.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This principle, of applying a given static analysis on each version of the program as a whole, gives us a new axis of analysis “for free”, namely that along the history of the program’s construction. By “for free”, we mean that we need not write any code: we simply take an existing tool for static analysis and run it at each step of the program’s construction. One might call this principle the naive approach to step by step program analysis.&lt;/p&gt;

&lt;p&gt;In addition to this naive approach, we could consider approaches to step-wise static analysis which try to utilize both the structured nature of our editor and the stream of change-operations it produces. Such approaches to static analysis will take as input:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;a previous version of the program,&lt;/li&gt;
  &lt;li&gt;a finished analysis for that previous version of the program and&lt;/li&gt;
  &lt;li&gt;a single change to the program (or part of it).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There are two important reasons to do so:&lt;/p&gt;

&lt;p&gt;The first is that the approach of running a full static analysis for each change to a program, might be “free” in terms of the effort to develop the analyses, but is certainly not free in terms of processing power! When we take this approach the amount of required processing power and time is multiplied by the number of changes we wish to analyze. This likely excludes many interesting use-cases, such as the one of interactive feedback on the output of static analyses.&lt;/p&gt;

&lt;p&gt;A second reason to consider a more incremental approach to static analysis is that such analyses, by virtue of their incremental formulation, may occasionally lead to output that is more easily understood by the human end-user of the analysis. In such analyses we spend quite some effort in precisely defining how various changes may affect the output of an analysis, and how information flows through a program, with the goal of reducing the analyses’ runtimes. It turns out this has the fortunate side-effect of saving processing power of the human interpreter of the result as well!&lt;/p&gt;

&lt;p&gt;In the past month I have written a number of such incremental static analyses; I will present the results here as a &lt;a href=&quot;/static-analysis/&quot;&gt;series of articles&lt;/a&gt;. In the &lt;a href=&quot;/l-a-toy-language/&quot;&gt;next article&lt;/a&gt; in the series we will define a small language, called &lt;em&gt;L&lt;/em&gt;, as a basis for various analyses.&lt;/p&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/static-analysis-intro/&quot;&gt;Static Analysis - Introduction&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on November 15, 2017.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Constraints on Replace]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/constraints-on-replace/" />
  <id>http://www.expressionsofchange.org/constraints-on-replace</id>
  <published>2017-11-14T00:00:00+01:00</published>
  <updated>2017-11-14T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;Let’s start with some quick pointers to important background information.&lt;/p&gt;

&lt;p&gt;The context of this article is &lt;a href=&quot;/introducing-the-editor/&quot;&gt;an editor of s-expressions&lt;/a&gt; in which the primitives are the mechanisms of construction rather than the resulting structures.&lt;/p&gt;

&lt;p&gt;Important examples of such primitives are the operations (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Delete&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt;). These primitives may be composed; a list of them is called a history.&lt;/p&gt;

&lt;p&gt;The operation &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt;, in particular, replaces a child-node with a new one, where the new node is described recursively using a full history &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt;; the child to be replaced is denoted by an index &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;i&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;In this article we will consider a possible constraint on this history &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt; and its ramifications.&lt;/p&gt;

&lt;h2 id=&quot;a-continuation-of-history&quot;&gt;A continuation of history&lt;/h2&gt;

&lt;p&gt;The potential constraint that we will consider is the following: that &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt; must be continuation of the history of the node previously existing at index &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;i&lt;/code&gt;. We shall call this constraint “&lt;em&gt;continuation-only&lt;/em&gt;” and cases in which it does not apply “&lt;em&gt;any-history&lt;/em&gt;”.&lt;/p&gt;

&lt;p&gt;Let’s make this notion of history being a continuation of a previously existing history a bit more precise.&lt;/p&gt;

&lt;p&gt;First, note that we can always meaningfully talk about such a thing as “the history of the node at index &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;i&lt;/code&gt;”: This is because the only way to add or modify children in the first place is by using the operations &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt;, and both of these operations take a history as an argument. All we need to do is, during construction of each child, keep a reference to the history out of which it was constructed. In the below, we will refer to such a pre-existing history as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h-pre&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;So what is the relationship between &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h-pre&lt;/code&gt; if a history is a continuation of another? As noted in the above, a history is simply a list of operations of change. Given that definition, we may formulate &lt;em&gt;continuation&lt;/em&gt; as follows: A history &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt; is a continuation of another history &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h-pre&lt;/code&gt; if the latter is a prefix of the former. In other words: the list &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt; fully contains the list &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h-pre&lt;/code&gt; at its beginning.&lt;/p&gt;

&lt;p&gt;An alternative, equivalent, formulation, is based on the following implementation detail for histories: In practice, we implement histories using a technique called &lt;a href=&quot;/hash-consing/&quot;&gt;hash consing&lt;/a&gt;. In short: the list is implemented as a linked list, where each element points to its predecessor using a hash. In this view, we can say that a history &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt; is a continuation of another history &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h-pre&lt;/code&gt; if you, by following the hashes from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt;, at some point will encounter the hash &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h-pre&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;To clarify what history continuation means in practice, let’s consider two examples:&lt;/p&gt;

&lt;h3 id=&quot;continuation-only&quot;&gt;Continuation-only&lt;/h3&gt;

&lt;p&gt;In the first example the history with which we replace is a continuation of the previously existing history:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 ...&lt;/code&gt;
    &lt;ul&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 (BecomeAtom *))&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 1 (BecomeAtom 6))&lt;/code&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace 0 ...&lt;/code&gt;
    &lt;ul&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 (BecomeAtom *))&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 1 (BecomeAtom 6))&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Delete 1&lt;/code&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;(The notation of nested lists in the above should be interpreted as follows: each of the ellipses (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;...&lt;/code&gt;) are a single parameter &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt;, which is further detailed in the indented bullet points below it.)&lt;/p&gt;

&lt;p&gt;This is an example of a continuation of history, because the first 3 operations, up until &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 1 (BecomeAtom 6))&lt;/code&gt; are shared between the pre-existing history &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h-pre&lt;/code&gt; and the new history &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt;.&lt;/p&gt;

&lt;h3 id=&quot;any-history&quot;&gt;Any-history&lt;/h3&gt;

&lt;p&gt;In the second example the history with which we replace is &lt;em&gt;not&lt;/em&gt; a continuation of the pre-existing one.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 ...&lt;/code&gt;
    &lt;ul&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 (BecomeAtom *))&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 1 (BecomeAtom 6))&lt;/code&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace 0 ...&lt;/code&gt;
    &lt;ul&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 (BecomeAtom +))&lt;/code&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is an example where the constraint does not apply: the operations &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 (BecomeAtom *))&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 1 (BecomeAtom 6))&lt;/code&gt; from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h-pre&lt;/code&gt; are not reflected in the history &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;In the below, we’ll consider the consequences enforcing &lt;em&gt;continuation-only&lt;/em&gt; from two perspectives: the perspective of the mental model of our programmers, and the perspective of potential simplifications and optimizations we can make to our tooling. We’ll start with the human perspective.&lt;/p&gt;

&lt;h3 id=&quot;continuation-only-mental-model&quot;&gt;Continuation-only: Mental model&lt;/h3&gt;

&lt;p&gt;Enforcing the constraint &lt;em&gt;continuation-only&lt;/em&gt; makes for a much more simple mental model of the nodes and histories under consideration than the alternative.&lt;/p&gt;

&lt;p&gt;To see why this is the case, consider the following: One of the reasons &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; is a separate operation in the first place is that it allows for a &lt;a href=&quot;/t-indices/&quot;&gt;semantics of identity in the face of change&lt;/a&gt;. In particular: we can talk about the node which is “replaced” as a single thing which exists over time, interpreting the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; as a change to its current value and history, but not to its fundamental identity.&lt;/p&gt;

&lt;p&gt;The constraint &lt;em&gt;continuation-only&lt;/em&gt; fits nicely into that model. If the constraint applies, the node to which we attach this identity has a single linear history. At times, new things may happen in that history, but those things that have once happened will never “unhappen”.&lt;/p&gt;

&lt;p&gt;This mental model also maps quite nicely to a typical use case: that of editing a document. Each time a part of the document is edited, that part’s history is simply extended with the edit-operation.&lt;/p&gt;

&lt;p&gt;Contrast this model with the alternative, being  &lt;em&gt;any_history&lt;/em&gt;. In that scenario, each time a node is replaced, its history may be arbitrarily rewritten.&lt;/p&gt;

&lt;p&gt;In such a scenario we cannot speak of &lt;em&gt;the&lt;/em&gt; history of any of our children, since the view of what that history is may itself change over time. Because of that, we must, for each of a node’s children, distinguish between one of two different histories:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;The history &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt; as set in the latest &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; for the child.&lt;/li&gt;
  &lt;li&gt;The history of such histories.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Such a distinction makes reasoning about a node and the histories of its subnodes quite a bit more complex!&lt;/p&gt;

&lt;h3 id=&quot;continuation-only-t-paths&quot;&gt;Continuation-only: T-paths&lt;/h3&gt;

&lt;p&gt;In an editor which takes operations of change as its primitives we get the ability to reference the children of any given node by their order of appearance “for free”. This is quite useful, because such a reference will keep pointing at the same child over time. This was discussed in depth in &lt;a href=&quot;/t-indices/&quot;&gt;a separate article&lt;/a&gt;, in which we called such references &lt;em&gt;t-indices&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;In that article we also mentioned &lt;em&gt;t-paths&lt;/em&gt;. The interpretation of such a &lt;em&gt;t-path&lt;/em&gt; is to consider it to be a list of &lt;em&gt;t-indices&lt;/em&gt;; to follow the path means to consider the elements of the path one by one and descend into the relevant child accordingly. As such a &lt;em&gt;t-path&lt;/em&gt; can point to an arbitrary descendant, not just a child.&lt;/p&gt;

&lt;p&gt;The possible utility of both &lt;em&gt;t-indices&lt;/em&gt; and &lt;em&gt;t-paths&lt;/em&gt; lies in the fact that, for some structure, they give us a reference to a descendant that keeps pointing at “the same” descendant in the face of changes to the structure over time. We could say that it is this utility that makes it so that the &lt;em&gt;t-index&lt;/em&gt; or &lt;em&gt;t-path&lt;/em&gt; does what you expect.&lt;/p&gt;

&lt;p&gt;For &lt;em&gt;t-indices&lt;/em&gt;, whether they do what you’d expect holds for both the cases &lt;em&gt;continuation-only&lt;/em&gt; and &lt;em&gt;any-history&lt;/em&gt;. This is because, in a single node with a single linear history, the assignment of t-indices to its children is only influenced by the occurrences of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert&lt;/code&gt; in that history. The interpretation of the attribute &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt; of any given &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; operation does not come into play at all. By extension, neither does any constraint on &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt;’s arguments.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;T-paths&lt;/em&gt;, however, only do what you expect for the case &lt;em&gt;continuation-only&lt;/em&gt;. This is perhaps best shown with a counter-example. Consider again the history as provided at the beginning of this article of an &lt;em&gt;any-history&lt;/em&gt;, which is shown here annotated with the resulting &lt;em&gt;t-paths&lt;/em&gt;:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 ...&lt;/code&gt; –  inserts a new node at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;t-index = 0&lt;/code&gt;
    &lt;ul&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 (BecomeAtom *))&lt;/code&gt;  –  resulting &lt;em&gt;t-path:&lt;/em&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0 0&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 1 (BecomeAtom 6))&lt;/code&gt;  –  resulting &lt;em&gt;t-path:&lt;/em&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0 1&lt;/code&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace 0 ...&lt;/code&gt;
    &lt;ul&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 (BecomeAtom +))&lt;/code&gt;  –  resulting &lt;em&gt;t-path:&lt;/em&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0 0&lt;/code&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;As should be obvious from the above, the &lt;em&gt;t-path&lt;/em&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0 0&lt;/code&gt; does not constitute a meaningful reference to a node with a single identity: it initially points at the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;*&lt;/code&gt;, and later at the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;+&lt;/code&gt;, despite these two nodes being unrelated entirely: they exist in separate histories.&lt;/p&gt;

&lt;p&gt;When the constraint &lt;em&gt;continuation-only&lt;/em&gt; applies, it is not possible to construct such a counter-example: in that scenario it is not possible to replace a history with an entirely unrelated one by definition.&lt;/p&gt;

&lt;h3 id=&quot;algorithms&quot;&gt;Algorithms&lt;/h3&gt;

&lt;p&gt;Further consequences &lt;em&gt;continuation-only&lt;/em&gt; may be found in the algorithms that make use of histories of operations.&lt;/p&gt;

&lt;p&gt;What could such algorithms be? First, &lt;a href=&quot;/constructing-s-expressions/&quot;&gt;there is the algorithm that constructs an actual s-expression out of a history of changes&lt;/a&gt; – something we will inevitably be interested in at some point. When compared to an editor that stores s-expressions directly, such an algorithm forms a computational overhead by default. If only for that reason, it is worthwhile to consider its performance characteristics.&lt;/p&gt;

&lt;p&gt;First, let’s consider a naive implementation: For each &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; operation, construct the resulting structure from scratch by considering each element of the history in turn. Such an approach is not scalable: Each &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; operation takes a full history as one of its parameters and such histories grow linearly in time. Moreover, the histories are recursively composed of further &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; operations. The result is an exponential growth of the required runtime.&lt;/p&gt;

&lt;p&gt;A less naive approach is the following: make use of the fact that each &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; operations replaces an already existing node, which already has some structure. If we can use this existing structure in such a way that we don’t have to consider the full provided history each time we replace, we may reduce the required runtime considerably.&lt;/p&gt;

&lt;p&gt;If &lt;em&gt;continuation-only&lt;/em&gt; applies, this is quite straightforward: we can simply take the existing structure as our starting point, and apply to it only those operations in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;h&lt;/code&gt; that are new. Such an approach is only linear in time with the amount of new operations.&lt;/p&gt;

&lt;h2 id=&quot;any-history-1&quot;&gt;&lt;em&gt;any-history&lt;/em&gt;&lt;/h2&gt;

&lt;p&gt;Finally, let’s considert the respective drawbacks and advantages of the alternative to &lt;em&gt;continuation-only&lt;/em&gt;, namely those of &lt;em&gt;any-history&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;The drawbacks should be obvious: they are precisely the inverse of the aformentioned advantages. In particular: the mental model of what &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; means becomes more complex; arbitrary descendants may no longer be referenced meaningfully using a &lt;em&gt;t-path&lt;/em&gt; and performant algorithms for construction of nodes are less obvious.&lt;/p&gt;

&lt;p&gt;Having said that, there is still something to be said for this approach: Relaxing the constraint gives us certain features almost by definition. Namely: the ability to rewrite the histories of children-nodes, while retaining, at the level at which the rewriting is done, a reference to the history as it existed before it was rewritten. This means we get both the advantages of rewriting of history (cleaner histories) and those of never rewriting history (no loss of information).&lt;/p&gt;

&lt;p&gt;To see how this is so, consider the example with which we started this article:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 ...&lt;/code&gt;
    &lt;ul&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 (BecomeAtom *))&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 1 (BecomeAtom 6))&lt;/code&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace 0 ...&lt;/code&gt;
    &lt;ul&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 (BecomeAtom +))&lt;/code&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In the above, from the perspective of the node whose history is rewritten, all that ever happened is the insertion of the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;+&lt;/code&gt;. However, from the root node we can still see that at some point there was a different history for that node.&lt;/p&gt;

&lt;p&gt;Note that this ability to rewrite history is &lt;em&gt;integrated&lt;/em&gt; into our main means of expression: we do not require a separate layer of abstraction for it. One caveat does apply though: it is only available on items that are at least one level lower in the tree. This also implies that rewriting history is not available for the root of the tree.&lt;/p&gt;

&lt;p&gt;It remains to be seen whether this integrated approach to rewriting of history should be seen as a useful means of expression, or an undesirable conflation of ideas.&lt;/p&gt;

&lt;h3 id=&quot;meaningful-rewrites&quot;&gt;Meaningful rewrites&lt;/h3&gt;

&lt;p&gt;That rewriting history is useful in the first place should perhaps be made clear with a separate example.&lt;/p&gt;

&lt;p&gt;The running example of &lt;em&gt;any-rewrite&lt;/em&gt; has been to present a rewriting of history that is truly arbitrary: in the example above there is truly no relationship between the history pre and post rewrite.&lt;/p&gt;

&lt;p&gt;In practice, rewriting of history will often not be so arbitrary. For an example of this, consider the concept of &lt;em&gt;undo&lt;/em&gt;, which can be modeled by replacing the history of a node by a history in which the last operation is no longer present:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 ...&lt;/code&gt;
    &lt;ul&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 (BecomeAtom *))&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 1 (BecomeAtom 6))&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 2 (BecomeAtom 9))&lt;/code&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace 0 ...&lt;/code&gt;
    &lt;ul&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BecomeList&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 0 (BecomeAtom *))&lt;/code&gt;&lt;/li&gt;
      &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert 1 (BecomeAtom 6))&lt;/code&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;conclusions--final-thoughts&quot;&gt;Conclusions / final thoughts&lt;/h3&gt;

&lt;p&gt;In the above we’ve seen the respective advantages and disadvantages of either enforcing the constraint that histories must be append-only or not. A few final thoughts are below.&lt;/p&gt;

&lt;p&gt;First, as noted in the above, a naive approach to &lt;em&gt;t-paths&lt;/em&gt; does not yield meaningful results. It is however, even in environments in which nodes can be replaced by arbitrary histories, possible to come up with a scheme of referencing children that does in fact yield meaningful references across time. This scheme will be explained in a future article.&lt;/p&gt;

&lt;p&gt;Second, we noted that in &lt;em&gt;continuation-only&lt;/em&gt; a performant algorithm for construction is readily available, whereas for for &lt;em&gt;any-history&lt;/em&gt; it is not as obvious what such an algorithm should look like. We simply postulate here that such algorithms can still be devised. In particular it is possible to come up with algorithms that are linear in runtime with respect to the amount of history-rewriting.&lt;/p&gt;

&lt;p&gt;Finally: given the above advantages and disadvantages, we are left with a question: should we apply the constraint or not?  Or should we perhaps make a distinction between environments in which the constraint applies, and environments in which it doesn’t? Or should we perhaps have 2 distinct operations for &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt;, one for each scenario?&lt;/p&gt;

&lt;p&gt;These are, as it stands, open questions. The present version of the edtior is &lt;em&gt;any-history&lt;/em&gt;. The rationale is: this forces me to think about the annoying corner-cases which exist only if the constraint does not apply. However, this is certainly subject to change.&lt;/p&gt;


  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/constraints-on-replace/&quot;&gt;Constraints on Replace&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on November 14, 2017.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[T-Indices]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/t-indices/" />
  <id>http://www.expressionsofchange.org/t-indices</id>
  <published>2017-11-09T00:00:00+01:00</published>
  <updated>2017-11-09T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;p&gt;Let’s start with some terminology.&lt;/p&gt;

&lt;p&gt;An &lt;a href=&quot;https://en.wikipedia.org/wiki/S-expression&quot;&gt;s-expression&lt;/a&gt; is a notation for tree-structured data. Each s-expression is either an atom or a list of s-expressions. Using this recursive definition arbitrary tree-structures can be formed.&lt;/p&gt;

&lt;p&gt;Because it is the tree-like properties that we’re interested in in this article, we will use the terminology of trees (nodes, children, parents) in the below. We will mostly ignore atoms and talk almost exclusively of list-expressions, as these are the only non-trivial case, being the only type of node that can have children.&lt;/p&gt;

&lt;p&gt;If we want to talk about a particular child of a node, a straightforward way of doing so is to reference it by its &lt;a href=&quot;https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html&quot;&gt;zero-based&lt;/a&gt; index in the list of children. I.e. in s-expression &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(a b)&lt;/code&gt; we can say that the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;a&lt;/code&gt; is the child at index &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0&lt;/code&gt;, and the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;b&lt;/code&gt; is the child at index &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;1&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This observation can be generalized to refer to descendants at arbitrary depth by using a list of such indices and interpreting it as a path through the tree. I.e. in the s-expression &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(a (b ((c d) e)))&lt;/code&gt; the atom &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;c&lt;/code&gt; can be reached by following the path &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;1, 1, 0, 0&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;In this approach we use the &lt;em&gt;spatial&lt;/em&gt; layout of the tree-structure to refer to its elements. It is for this reason that we will refer to such indices as &lt;em&gt;s-indices&lt;/em&gt; in the below.&lt;/p&gt;

&lt;h3 id=&quot;t-indices&quot;&gt;T-indices&lt;/h3&gt;

&lt;p&gt;In the editor presented in this project we &lt;a href=&quot;...&quot;&gt;treat the modifications to s-expressions as first level citizens&lt;/a&gt;. The central motivation for doing this is the idea that changes to computer programs are such a central aspect of the programming experience that they must be put more centrally in the tooling.&lt;/p&gt;

&lt;p&gt;We do this by offering a small set of operations for constructing s-expressions, and storing those means of construction rather than the constructed structures. Key elements of this set of operations are &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Delete&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Given these operations, a second way of referring to the children of a node becomes available: by referring to them in their order of appearance. In other words: when were they inserted?&lt;/p&gt;

&lt;p&gt;Consider the following full history of operations on a list node:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Insert &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;b&lt;/code&gt; at s-index 0.&lt;/li&gt;
  &lt;li&gt;Insert &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;a&lt;/code&gt; at s-index 0.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Constructing a node using these operations will yield the expression &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(a b)&lt;/code&gt;, given that insertion at a given s-index will shift any items at or above that s-index one place to the right&lt;/p&gt;

&lt;p&gt;In a such constructed node, we may refer to the node &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;b&lt;/code&gt; as created at &lt;em&gt;t=0&lt;/em&gt; and the node &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;a&lt;/code&gt; as created at &lt;em&gt;t=1&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Because we we look at the node through the lens of &lt;em&gt;time&lt;/em&gt; as opposed to its current &lt;em&gt;spatial&lt;/em&gt; layout in this approach, we’ll denote this indexing scheme as the &lt;em&gt;t-index&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Note that the same generalization to paths to arbitrary descendants is available for t-indices as it was in the case of regular indices, &lt;a href=&quot;/constraints-on-replace/&quot;&gt;although some constraints must apply on replace’s paramters for the path to be stable over time&lt;/a&gt;.&lt;/p&gt;

&lt;h4 id=&quot;delete&quot;&gt;Delete&lt;/h4&gt;

&lt;p&gt;Referring to children in order of insertion raises the question: how does this interact with some of the other main operations, namely &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Delete&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt;?&lt;/p&gt;

&lt;p&gt;The answer for &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Delete&lt;/code&gt; is straightforward: deleting a child makes it go away in the current version, but does not alter the order of appearance. Consider the following full history of a node:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Insert &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;b&lt;/code&gt; at s-index 0.&lt;/li&gt;
  &lt;li&gt;Insert &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;a&lt;/code&gt; at s-index 0.&lt;/li&gt;
  &lt;li&gt;Delete the item at s-index 1.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Constructing a node using these operations yields an expression &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(a)&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;At t-index 1 we still have the node &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;a&lt;/code&gt;, as in the previous example. At t-index 0, however, we now have nothing. This is because the child &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;b&lt;/code&gt;, which lived at (spatial) index 1 at the time of deletion, was the item at t-index 0. Such an answer “nothing” may be implemented using a special &lt;em&gt;tombstone&lt;/em&gt; or &lt;em&gt;sentinel&lt;/em&gt; value.&lt;/p&gt;

&lt;h4 id=&quot;replace&quot;&gt;Replace&lt;/h4&gt;

&lt;p&gt;How &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; operations affect t-indices is less straightforward – in the design of the editor, one actually has to make a choice between one of 2 options.&lt;/p&gt;

&lt;p&gt;As a first attempt, we could simply view Replace as a shorthand for a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Delete&lt;/code&gt; followed by and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Insert&lt;/code&gt;. In that model, after a replace, the t-index previously pointing to the now-replaced child will yield the &lt;em&gt;tombstone&lt;/em&gt;, and the new value for the child will be associated with a new consecutive t-index. Considering the following example:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Insert &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;b&lt;/code&gt; at s-index 0.&lt;/li&gt;
  &lt;li&gt;Insert &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;a&lt;/code&gt; at s-index 0.&lt;/li&gt;
  &lt;li&gt;Replace the item at s-index 1 with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;c&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The t-index 0 would, just like in the example for Delete, yield no result. The t-index 2 would point to the child &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;c&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;A second possible definition for the interplay between &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Replace&lt;/code&gt; and t-indices is quite different: In it, the t-index of the replaced child is reused to point to the new child after the replacement.&lt;/p&gt;

&lt;p&gt;In this definition, after the same set of operation as in the last example, the t-index 0 points to the item &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;c&lt;/code&gt; and the t-index 2 is as of yet unused.&lt;/p&gt;

&lt;p&gt;As such, we can use the t-index to answer the question “what is the child that was &lt;em&gt;nth&lt;/em&gt; element in the order of insertion, and may since have been changed any number of times?”. This means that we can use t-indices as a reference &lt;a href=&quot;/ship-of-theseus/&quot;&gt;to an identity that persists under mutation&lt;/a&gt;. In this model the identity is formed at insertion, and replace is interpreted as a mutation that does not alter identity.&lt;/p&gt;

&lt;p&gt;Note that in this model, in cases where modeling identity persisting across a replace is undesirable, we always have the option of explicitly deleting and inserting instead.&lt;/p&gt;

&lt;p&gt;It is this second definition that we have taken in the editor.&lt;/p&gt;

&lt;h3 id=&quot;uses-for-t-indices-in-the-ui&quot;&gt;Uses for t-indices in the UI&lt;/h3&gt;

&lt;p&gt;Associating an identity with nodes on the moment of their creation, and having a simple way (namely: consecutive numbers) to refer to them is useful in a number of ways.&lt;/p&gt;

&lt;p&gt;A first such use case is in presenting historical views to the user. Consider again the following 3 operations:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Insert &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;b&lt;/code&gt; at s-index 0.&lt;/li&gt;
  &lt;li&gt;Insert &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;a&lt;/code&gt; at s-index 0.&lt;/li&gt;
  &lt;li&gt;Delete the item at s-index 1.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It is not immediately obvious that the deletion operates on the item &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;b&lt;/code&gt;, which has since its creation been shifted to s-index 1 because another item was inserted in its place. This is the case even for this trivial example; in any non-trivial example this lack of clarity renders historical views which only display an s-index quite useless from a UI perspective.&lt;/p&gt;

&lt;p&gt;To remedy this, we can use t-indices in our UI, and decorate the above to the show the following:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Insert &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;b&lt;/code&gt; at s-index 0, yielding t-address 0.&lt;/li&gt;
  &lt;li&gt;Insert &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;a&lt;/code&gt; at s-index 0, yielding t-address 1.&lt;/li&gt;
  &lt;li&gt;Delete the item at s-index 1 (which is t-address 0)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Of course, in an actual UI we are not restricted to such a rather clumsy textual notation. Instead, we could e.g. use colors to highlight the relationships between various operations at the same t-address.&lt;sup id=&quot;fnref:t-addresses-everywhere-is-not-possible&quot;&gt;&lt;a href=&quot;#fn:t-addresses-everywhere-is-not-possible&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;The idea of a node having an persistent identity in the context of changes to its value is useful more generally. One example is in an interface which has multiple views on a single document, e.g. in the form of multiple tabs or windows. In such a setup one can imagine certain windows displaying part of a larger tree; and edits to this part of the tree being consistently propagated from and to other windows that show the same part of the tree. The idea of sameness can then be modeled using t-paths.&lt;/p&gt;

&lt;p&gt;In particular, this means one can edit part of a tree in one separate window, without any possible confusion as to what it means if that part is moved around in the document. (Such moving around may occur as a result of insertions and deletions in its parent node). In cases where the editor has only a single user and a single thread of control, such confusion is ruled out entirely. In distributed and multi-user contexts we are able to significantly reduce the amount of such confusion (“merge conflicts”); more about which in a separate article.&lt;/p&gt;

&lt;h3 id=&quot;conclusions&quot;&gt;Conclusions&lt;/h3&gt;

&lt;p&gt;T-indices, and by extension t-paths, are a useful tool to unambiguously talk about a particular node in a mutable tree.&lt;/p&gt;

&lt;p&gt;They require a historical view of the tree, but given such a view are quite trivially implemented.&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:t-addresses-everywhere-is-not-possible&quot;&gt;
      &lt;p&gt;One might be tempted to take t-addresses as the basis of modeling of the 3 operations if it is indeed a useful representation. Note however that for insertions we always need the s-address, and hence such a model would lead to an inconsistant mix of s-addresses and t-addresses in the operations. &lt;a href=&quot;#fnref:t-addresses-everywhere-is-not-possible&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/t-indices/&quot;&gt;T-Indices&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on November 09, 2017.&lt;/p&gt;</content>
</entry>


<entry>
  <title type="html"><![CDATA[Git is ignorant by design]]></title>
  <link rel="alternate" type="text/html" href="http://www.expressionsofchange.org/git-is-ignorant-by-design/" />
  <id>http://www.expressionsofchange.org/git-is-ignorant-by-design</id>
  <published>2017-11-08T00:00:00+01:00</published>
  <updated>2017-11-08T00:00:00+01:00</updated>
  <author>
    <name>Klaas van Schelven</name>
    <uri>http://www.expressionsofchange.org</uri>
    <email>klaas@vanschelven.com</email>
  </author>
  <content type="html">&lt;h3 id=&quot;an-example&quot;&gt;An example&lt;/h3&gt;

&lt;p&gt;The concept is best explained using an example. Consider the following output of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;git show&lt;/code&gt;, which shows a change to a piece of Python code:&lt;/p&gt;

&lt;div class=&quot;language-diff highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;commit 9dadc2.........
Author: ......
Date: ......
&lt;/span&gt;&lt;span class=&quot;err&quot;&gt;
&lt;/span&gt;    Refactoring: move method_c and method_d
    
    These methods are moved from OneClass to AnotherClass, as the latter
    is the correct location because of some reasons that would certainly
    exist if this wasn&apos;t a toy example.
&lt;span class=&quot;err&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;gh&quot;&gt;diff --git a/example.py b/example.py
index 1cde8fb..f487c7e 100644
&lt;/span&gt;&lt;span class=&quot;gd&quot;&gt;--- a/example.py
&lt;/span&gt;&lt;span class=&quot;gi&quot;&gt;+++ b/example.py
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;@@ -2,17 +2,17 @@&lt;/span&gt; class AnotherClass(object):
     def method_a(self):
         pass  # implementation of method_a ...
 
&lt;span class=&quot;gd&quot;&gt;-
-class OneClass(object):
-    def method_b(self):
-        pass  # implementation of method_b ...
-
&lt;/span&gt;     def method_c(self):
         pass  # implementation of method_c ...
 
     def method_d(self):
         pass  # implementation of method_d ...
 
&lt;span class=&quot;gi&quot;&gt;+
+class OneClass(object):
+    def method_b(self):
+        pass  # implementation of method_b ...
+
&lt;/span&gt;     def method_e(self):
         pass  # implementation of method_e
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;How should we interpret this diff?&lt;/p&gt;

&lt;p&gt;Let’s start by reading what the author ostensibly intended. The commit message talks about moving the two methods &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;method_c&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;method_d&lt;/code&gt; from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;OneClass&lt;/code&gt; to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AnotherClass&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;However, if we look at the actual diff, we see something else entirely: the removal of the definition of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;OneClass&lt;/code&gt; and its method &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;method_b&lt;/code&gt; in one place, and the reintroduction of these elements in another place.&lt;/p&gt;

&lt;p&gt;It might seem that either the commit message or the actual contents of the commit are wrong. It turns out that neither is the case. Instead, the source of confusion lies in the way &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;git&lt;/code&gt; works&lt;sup id=&quot;fnref:or-any-text-based-vcs&quot;&gt;&lt;a href=&quot;#fn:or-any-text-based-vcs&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot; role=&quot;doc-noteref&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;

&lt;h3 id=&quot;snapshots-on-text-files&quot;&gt;Snapshots on text files&lt;/h3&gt;

&lt;p&gt;Git views the world as a collection of files (usually: text files), of which snapshots are made each time a commit is made. This means that, for the example above, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;git&lt;/code&gt; got to see the following two versions:&lt;/p&gt;

&lt;h4 id=&quot;before-the-commit&quot;&gt;Before the commit:&lt;/h4&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;AnotherClass&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;object&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
     &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;method_a&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
         &lt;span class=&quot;k&quot;&gt;pass&lt;/span&gt;  &lt;span class=&quot;c1&quot;&gt;# implementation of method_a ...
&lt;/span&gt; 

&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;OneClass&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;object&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;method_b&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;pass&lt;/span&gt;  &lt;span class=&quot;c1&quot;&gt;# implementation of method_b ...
&lt;/span&gt;
     &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;method_c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
         &lt;span class=&quot;k&quot;&gt;pass&lt;/span&gt;  &lt;span class=&quot;c1&quot;&gt;# implementation of method_c ...
&lt;/span&gt; 
     &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;method_d&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
         &lt;span class=&quot;k&quot;&gt;pass&lt;/span&gt;  &lt;span class=&quot;c1&quot;&gt;# implementation of method_d ...
&lt;/span&gt; 
     &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;method_e&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
         &lt;span class=&quot;k&quot;&gt;pass&lt;/span&gt;  &lt;span class=&quot;c1&quot;&gt;# implementation of method_e
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h4 id=&quot;after-the-commit&quot;&gt;After the commit:&lt;/h4&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;    &lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;AnotherClass&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;object&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
         &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;method_a&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
             &lt;span class=&quot;k&quot;&gt;pass&lt;/span&gt;  &lt;span class=&quot;c1&quot;&gt;# implementation of method_a ...
&lt;/span&gt;     
         &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;method_c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
             &lt;span class=&quot;k&quot;&gt;pass&lt;/span&gt;  &lt;span class=&quot;c1&quot;&gt;# implementation of method_c ...
&lt;/span&gt;     
         &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;method_d&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
             &lt;span class=&quot;k&quot;&gt;pass&lt;/span&gt;  &lt;span class=&quot;c1&quot;&gt;# implementation of method_d ...
&lt;/span&gt;     

    &lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;OneClass&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;object&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;method_b&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
            &lt;span class=&quot;k&quot;&gt;pass&lt;/span&gt;  &lt;span class=&quot;c1&quot;&gt;# implementation of method_b ...
&lt;/span&gt;
         &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;method_e&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
             &lt;span class=&quot;k&quot;&gt;pass&lt;/span&gt;  &lt;span class=&quot;c1&quot;&gt;# implementation of method_e
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;First note that these 2 versions actually correspond to the commit message: the 2 methods &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;method_c&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;method_d&lt;/code&gt; are indeed moved from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;OneClass&lt;/code&gt; to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AnotherClass&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Now, why does &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;git&lt;/code&gt; present a diff that seems to tell a different story? To answer this, we must first understand what &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;git diff&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;git show&lt;/code&gt; do in the first place: Given two versions of a file these commands attempt to calculate the smallest difference between them and present this.&lt;/p&gt;

&lt;p&gt;Regarding the meaning of the content of the files Git makes only a very minimal number of assumptions. Most of these have to do with “plain text files”. For example, it knows the special meaning of newline-characters. Using this knowledge the goal of finding a minimal difference is interpreted to mean “with a minimal number of different lines”.&lt;/p&gt;

&lt;p&gt;Given these goals, the diff as presented in the original example is exactly what you’d expect: removing &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;OneClass&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;method_b&lt;/code&gt; in one place and adding them in another is simply more efficient than the alternative of removing and adding &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;method_c&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;method_d&lt;/code&gt;. The former affects 5 removed lines and 5 added lines for a total of 10 while the latter affects 2 × 6 = 12 lines. (In both cases empty lines contribute to the numbers). Given that &lt;em&gt;Git&lt;/em&gt; attempts to show the smallest diff, it prefers the former options over the latter.&lt;/p&gt;

&lt;h3 id=&quot;the-problem-defined&quot;&gt;The problem defined&lt;/h3&gt;

&lt;p&gt;Given that &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;git diff&lt;/code&gt; behaves according to specification, what’s the problem? Can we make the suggestion that the original example is confusing more precise? In fact we can!&lt;/p&gt;

&lt;p&gt;The key to this is in realizing the following: Our example deals with a piece of Python code, in which concepts such as “classes” and “methods” exist. These concepts have a hierarchical nature: classes are structures that may contain other structures such as methods. When editing the file, we are aware of this structure – in fact, it forms the very reason we edit the file, an intent that the author even captured in the commit message.&lt;/p&gt;

&lt;p&gt;Moving part of a hierarchical structure to a different container is quite a useful and natural thing to be able to express. For comparison, consider the following phrases: “moving files to another directory” and “moving marbles to a different vase”.  Contrast this with the unnatural alternative of saying that the containers have, so to speak, “wrapped around newly contained items”: We don’t usually say “we’ve moved a directory-ending somewhat, so that it now contains 3 more files”, nor do we say “a vase has been moved to a location where it happens to contain more marbles”.&lt;/p&gt;

&lt;p&gt;Git, however, occasionally does precisely that, as demonstrated in the original example: it moves a class definition relative to its contents rather than vice versa.&lt;/p&gt;

&lt;p&gt;This is because Git remains agnostic of the structure of the files under its control, noting only that they are text files. Given that it can simply do no better than calculating the smallest difference in number of changed lines. This smallest number of lines happens to include the class definition, rather than the lower-level methods contained therein.&lt;/p&gt;

&lt;p&gt;It is for this reason that we can say “Git is &lt;em&gt;ignorant&lt;/em&gt;” – it’s ignorant of the structure of the files it’s diffing.&lt;/p&gt;

&lt;h3 id=&quot;closing-remarks&quot;&gt;Closing remarks&lt;/h3&gt;

&lt;p&gt;This concludes the main discussion of the problem. Some final thoughts are below.&lt;/p&gt;

&lt;p&gt;First, one might wonder how serious of a problem this really is. On this I can only speak from personal experience: I personally run into some version of this problem approximately once a week. Admittedly, the most regular occurrences are also less confusing than the example at the top of this article, which was contrived specifically to be maximally confusing.&lt;/p&gt;

&lt;p&gt;A more regular occurrence is a diff like the below:&lt;/p&gt;

&lt;div class=&quot;language-diff highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;gd&quot;&gt;--- a/example.py
&lt;/span&gt;&lt;span class=&quot;gi&quot;&gt;+++ b/example.py
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;@@ -1,3 +1,8 @@&lt;/span&gt;
 @decorated_function
&lt;span class=&quot;gi&quot;&gt;+def newfunction():
+    pass  # this is newfunction
+
+
+@decorated_function
&lt;/span&gt; def oldfunction():
     pass  # this is oldfunction
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;In the above example, the more logical diff would identify the first of the 2 decorators (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;@decorated_function&lt;/code&gt;) as being the newly added one. However, git cannot distinguish between the 2 identical occurrences of that line, and identifies the second one as the new one.&lt;/p&gt;

&lt;p&gt;If we read this diff like a story, git seems to tell us the following: the decorator of the function &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;oldfunction&lt;/code&gt; was repurposed for a newly added function &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;newfunction&lt;/code&gt;. The function &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;oldfunction&lt;/code&gt; was then given a new (identical) decorator to compensate for its loss. A simpler and more correct story would be: a new function &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;newfunction&lt;/code&gt; was added with its own decorator.&lt;/p&gt;

&lt;p&gt;A final example uses lists, in which it would be more logical to show the earlier opening bracket as the new addition. Such an approach would lead to a diff with balanced brackets, as opposed to the close-then-open we get now:&lt;/p&gt;

&lt;div class=&quot;language-diff highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;gd&quot;&gt;--- a/matrix.py
&lt;/span&gt;&lt;span class=&quot;gi&quot;&gt;+++ b/matrix.py
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;@@ -5,6 +5,11 @@&lt;/span&gt; matrix = [
         3,
     ],
     [
&lt;span class=&quot;gi&quot;&gt;+        7,
+        8,
+        9,
+    ],
+    [
&lt;/span&gt;         4,
         5,
         6,
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;In short: examples of the problem are easily found. However: the most regular occurrences are also generally the least confusing.&lt;/p&gt;

&lt;p&gt;As of yet, I have not been able to find other people reporting a similar experience. Whether this is because others don’t experience this phenomenon as a problem, or whether they have resigned themselves to Git’s set of assumptions, I do not know. You are invited let me know, possibly by presenting any of your own real-life examples in the comment section below.&lt;/p&gt;

&lt;h3 id=&quot;heuristics-and-fundamental-solutions&quot;&gt;Heuristics and fundamental solutions&lt;/h3&gt;

&lt;p&gt;As noted in the above, git’s ignorance is a fundamentally unsolvable problem: diffs that are based on text alone can never fundamentally know what the underlying structures of those text files were. This does not mean that finding better heuristics for presenting diffs is entirely impossible.&lt;/p&gt;

&lt;p&gt;The option &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;--indent-heuristic&lt;/code&gt;, introduced in Git version 2.11, attempts to do just that: it uses indentation as a heuristic for hierarchical structure; this corresponds to the reality in many programming languages and data formats.&lt;/p&gt;

&lt;p&gt;However, for obvious reasons, a solution using heuristics can only lead to limited results. Thoughts on more fundamental solutions which require no heuristics at all are the subject of the rest of this website.&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:or-any-text-based-vcs&quot;&gt;
      &lt;p&gt;Git, being the most popular Version Control System, is singled out here; most remarks in this article apply in fact to any modern VCS that ignores the underlying structure of the (text) files under its control. &lt;a href=&quot;#fnref:or-any-text-based-vcs&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;

  &lt;p&gt;&lt;a href=&quot;http://www.expressionsofchange.org/git-is-ignorant-by-design/&quot;&gt;Git is ignorant by design&lt;/a&gt; was originally published by Klaas van Schelven at &lt;a href=&quot;http://www.expressionsofchange.org&quot;&gt;Expressions of Change&lt;/a&gt; on November 08, 2017.&lt;/p&gt;</content>
</entry>

</feed>