An Exemplar on Coding-as-story | Articles | Ant Smith

I wrote a couple of articles developing the idea of using narrative to ignite cognition in respect of understanding software source code.

There's a whole bunch of background detail (quotes and references) in those other articles so I won't repeat it all here. Suffice to say: knowing isn't understanding. To understand, knowledge must be experienced. Story is a core cognitive function that does exactly that - ignites cognition, lets us experience what we are hearing or reading. Obviously not in terms of touch, taste, smell - but certainly in terms of feeling.

Knuth was right, we ought to be able to curl up with our source code as though it were a novel.

My new paradigm (Narratival-Exposition) aims to achieve that; the transposition of code and code-architecture into plot points and narrative.

My initial thoughts (especially the proposed grammar) were quite complex and left me wondering "but is it doable, what would it be like creating software in this way?"

So I gave it a bash. I wrote an application from scratch applying - at least some aspects of - the expositional grammar; just to see what I would get, and how the approach might help not only others to experience the code, but myself as I developed it. The app I chose was a set of Python scripts that would extract its own story (or in fact, extract story from any code written using the grammar).

This article is all about what happened, what I got out of it, and how it felt

Groundworks

Of course I live in different times to Knuth, who had to do all the hard work around just getting his code-novels actually readable (in terms of presentation with his TeX development) - these days we have presentation reasonably cracked with HTML and CSS.

The latter 2 criticisms are kind of out-of-scope for me; typesetting is (I believe) an obviated concern, and adoption is an issue for another time... We'll see later how these other concerns are avoided in the NeP (Narratival-Exposition Paradigm) approach, but for me the greatest criticism of Literate Programme is simply... the stories told are boring! They do not excite cognitive ignition. E.g. to quote from the WEB story: "'font_info' contains data about fonts."

Core to NeP is the application of the grammar of story-telling; which is told in terms of rich metaphor.

We utilise THROUGHLINEs to connect metaphor to intent. For this font_info example we perhaps would have first seen:

And then later when we needed to declare a font_info type construct we might see

I'll talk more about metaphor; it is foundationally important to the paradigm.

In summary, the NeP approach specifically addresses criticisms of Structure, Dissonance and Completeness in Literate Programming, whilst adding a rich story grammar in support of metaphorical story telling.

For more context, you can take a look at the source code for the extraction of story (from source code) over on GitHub: code-as-story - along with the output in the expo.md markdown file.

Extracting Story

Before I talk more about building the story, I ought to explain the extraction process, from a usage point-of-view...

This is a nascent proof-of-concept, so there's no real workflow support.

Of course, if you re-run the process there's a danger of overwriting the editorialised txt-file, but that is just a weakness of the tooling scripts that could be easily remedied in a production version of the scripts.

This editorial stage though is a critical, human, endeavour. This is where we make the story structure orthogonal to the code architecture. It turns the individual parts-of-story into a plot. This distinction between story and plot is key in any narrative construction (novel, screenplay or code base); and is one of the things missing in the original Literate Programming approach.

The parts of story are woven into the code, living with the pieces of code they describe - not as a separate document - and are therefore every bit as maintainable as a more traditional 'docstring' approach to documentation. In fact, the expositional blocks of the grammar can happily co-exist with traditional code-architecture based commentary. The narrative is an extra dimension, not a replacement for any needed technical observances.

The narrative-arc of the story (in the editorialised txt-file) is a separate document and does impose a degree of overhead. It's important that the story can have its own telling (a structure orthogonal to the code-architecture), so I deliberately resisted encoding anything into the code base that directed the narrative. The code base holds the parts of story; a separate file directs the narrative. The good news is, the narrative is pretty skeletal. Generating it is a matter off re-ordering the discovered lexical references, and grouping them into sections (adding headings). It should not be necessary to write large blocks of text in to the narrative file.

Principles of story-coding

The number 1 principle is that, rather than looking to describe code, we look to re-imagine the problem-at-hand as some metaphorical story that we can encode and later narrate.

Although much of the story stands-out as blocks of exposition within the code, we should find the chosen metaphor reflected in the code itself; most especially in the names of things within the code statements. It isn't enough to add a comment that says 'font_info in the next line is like a garment selected from our wardrobe'. Having selected 'garments in a wardrobe' as our metaphor (as expressed in the THROUGHLINE) we then code-up objects that are called garment, or wardrobe.

At one point, I thought that it would be important to choose a singular metaphor that could stand throughout a codebase. But actually, working in the paradigm, I think this is impossible, unhelpful and unnecessary. What really matters, is to avoid mixed-metaphor in a given context.

Some concepts jump-metaphor quite happily. For example, I found that my 'record_history' method name worked in both my REGISTRAR (lineage metaphor) class and my GRANULATOR (manufacturing process metaphor) class. Other concepts can be aliased quite easily. In the manufacturing metaphor symbolic tokens were prescribed a GrainType enumeration. The lexicographer imported the same enumeration as LexicalCategory. Where the metaphorical shift was more significant other methods are needed. For example, in the manufacturing stage the Grain class was given a 'semantics' method; providing an alternate view of a grain (wherein the grain's substance is returned as its semantic content) suitable for the lexicographer's metaphor. I.e. dissonance between metaphors (and/or team members) is resolved through various forms of aliasing. Obvs. a team will, at times, need to collude on this. There perhaps ought to be a specific verb in the paradigm's grammar also.

Blocks of exposition always precede the thing they discuss (in my Python implementation I have allowed for decorators that also must precede that which they decorate). More traditional code documentation uses docstrings that sit inside the thing they document. This is a purposeful distinction. In the exposition we talk about the role this thing plays in the overall story, not about its internal mechanics; thus, the exposition s its above and before, not within.

In my original proposed grammar, I thought the THROUGHLINE would be a means to express how differing parts of story worked together. But honestly, in practice, that is too complex AND has the effect off trying to weave narrative into the code; which I don't think is ever going to work - the code should express parts-of-story with any narrative impacts woven externally. Instead, I realised, the strength of the THROUGHLINE lies in interfacing the chosen metaphor to the wider context; which is exactly how I have used it.

I also, initially thought, that all the different verbs in the paradigm's grammar would give rise to different automated extractions. The practicality is that very little is needed in those terms. PROSE exposition has special handling, and CONTINUUM exposition could do with special handling, but otherwise associating an exposition to the following lexical identity works quite well. The real benefit of the various verbs (e.g. BEHAVIOUR versus MECHANISM or SKILL) helps us to understand the nature and extent of the code we are describing. E.g. BEHAVIOURs are typically a sequence (or iteration) of simpler 'actions'. A simpler 'action' maybe a SKILL (e.g. detection of disposition or transition) or a MECHANISM (e.g. a specific focussed update). Such distinctions are also useful when it comes to grouping expositions in the narrative, probably discussing each off the BEHAVOURs before looking at the SKILLs and MECHANISMs.

I'm all for having a rich vocabulary (since that helps us craft story) but defining the precise vocab is probably a per-project opportunity.

Oh! I should say, if you haven't read my earlier articles and have no idea what the paradigm's grammar actually is then take a look at the expo.md file in the above linked GitHub repository!

STORY-CODING!

So, what is it like, to write code like this?

First off, enhanced fun.

There's a real joy, sense of achievement, when you see a chosen metaphor come alive. When it meets the evolving needs of code - I hit a snag in my Granulator when trying to determine the active pythonic scope of an in-line comment. The expected DEDENTation symbols were arriving after the comment, even though the actual dedentation in the source code occurred before. Anyone else who has parsed Python code will recognise this behaviour; INDENTs and DEDENTs (as symbols, not as whitespace characters) relate only to actual code lines, not at all to any intervening comments; which isn't helpful to my use case. So, I had to jiggle the tokens around, ensuring that INDENT and DEDENT symbols would precede and in-line comments. I was a bit dismayed at first because I did not want to step out of the metaphor and start taking about switching the order of tokens. But thinking more about the metaphor I realised it was exactly as if my process-material had fine suspensions made up of COMMENT, INDENT and DEDENT particles and all I really needed was a fine mixing stage... so the metaphor was able to grow, quite naturally, in-line with the changing requirement; to my mind, a successful metaphor.

And it came with a strong sense of satisfaction. It felt good, and it felt right.

But does that even matter?

YES.

Yes, it does.

One thing you can be sure of, if it feels good, then it is good.

I also found, during coding, that I was adding small methods here and there specifically to maintain the metaphor. For example, in my SAMPLE(CODEX) class I added the one-line method 'particle_name' which simply calls the base CODEX class's 'token_val' method. This ensured that code in the manufacturing metaphor could talk in terms of particles, without having to distribute references to 'token_val'. As well as strengthening the metaphor, this also aids in the GEP (good engineering practice) of the code in terms of encapsulation.

It turns out that creating a story metaphor helps to refine the quality of the code for which the metaphor stands. I found this to be true, time and time again. Story-coding certainly helps the coder in terms of cognition; feeling just when it is right.

But will it also help others understand, not just know, a code base?

It's hard to tell without feedback but I do have a couple of qualms, in respect of this app and its metaphors.

First off, parsing language syntax is an old problem; one of the first software engineers had to address since it is a problem that lies at the heart of compilers and interpreters that make it possible to have computer languages! This means that most seasoned professionals would understand the narrator more readily if I had stuck to talking about token streams throughout...

Also, perhaps I am way more au-fait with the details of manufacturing processes than most; perhaps the GRANULATOR metaphor is just too niche? But then, software engineering has always benefitted at least as much from the super-generalist than the super-specialist. People who readily map domain to domain in varied problem spaces; people who take joy in learning the intricacies of alien worlds. For them a somewhat oblique metaphor might be just the ticket...

I think these are probably questions that can only be answered 'in the wild', through adoption. So perhaps, time will tell.

But, whatevs, I do think this is a strong evolution on the concepts of Literate Programming - and I will do more!!!