Background and Motivation

I’ve long been interested in Bret Victor’s Dynamicland and RealTalk work, but I’ve also found it frustrating that the system is entirely closed-source. He has explained his reasoning for this, but it does limit the ability to experiment with the ideas first hand.

The only publicly available system in a similar space that I could find was the Folk Computer project. Although I have a fondness for Tcl and Linux, I ultimately decided to build a more conventional Windows-based C# implementation instead.

Source code is available at https://github.com/ynformatics/TabulaLuma

To be clear, I have no inside knowledge of Bret Victor’s system. This project is based solely on interpretations of his published papers, talks, and demos. These are excellent by the way, and I recommend browsing through them to get a better idea of the thinking behind this work.

TL;DR;

  • Set up and calibrate the hardware as described below
  • Use the FrameCodeGenerator program to print out frames with program ids 0 and 1
  • Program 0 is a code editor, it will show a whisker pointing up. Move program 1 over the end of the whisker. You can now edit the code for program 1. Press Ctrl + S to save.
  • print out frames with program ids 2, 3 and 4. They should display something when placed on the table. Bring them in range of the code editor to see the code. This is decompiled from the assembly bytecode. You can edit and save the modified code which will generate an overriding .txt script file.

Core Concept

The central idea is to give physical objects programmable behaviour.

A canonical example is using sheets of paper tagged with fiducial markers. These markers allow the system to track each objects identity and position on a table. Programs can then be associated with individual pieces of paper, enabling text and graphics to be projected directly onto them using an overhead projector.

In effect, the physical paper becomes both an interface and a computational entity.

Reactive Programming Model

Programs in this system are written in a reactive style, built around three core concepts:

Claims

A Claim represents a statement of fact about the systems current state.

(page-1) has width 123.4

Claims describe what is.

Wishes

A Wish represents a desired state, something that should become true.

(page-1) is labelled "hello world!"

Wishes describe what should be.

Whens

A When defines the actions required to make a wish come true when certain conditions are met.

Conceptually:

When /page-id/ is labelled /label/, then draw the label on the corresponding page.

Whens form the bridge between declarative intent (wishes) and imperative behaviour (actions).

Implementing the Model in C#

Implementing this model in C#, a statically typed language, introduces some additional complexity. Dynamic languages such as Lua or Tcl are more naturally suited to this style, but it is still feasible in C# with careful design.

Claims and Wishes

Claims and wishes map cleanly to simple API calls:

Claim("(page-1) has width (223.4)");
Wish("(you) is labelled 'Hello World!'");

Whens with Actions

A When statement associates a pattern with an action. The action receives a binding context that provides typed access to the matched variables.

When("/page-id/ is labelled /label/", b => {    
    var id = b.Int("page-id");    
    var label = b.String("label");
    var ill = new Illumination();    
    ill.Text(label);
    Wish($"({id}) has illumination '{ill}'");
});

In this example:

  • The pattern extracts the page identifier and label text
  • The action constructs a visual representation
  • A new wish asserts that the page should display that illumination

Note that any valid C# code can be used in the action block and the full .Net framework is available. Claims/Wishes/Whens can be nested as required.

Handling the Non-Matching Case

A When can...

Read more »