Kurt Nørmark ©
Department of Computer Science
Aalborg University
Denmark

Abstract Index References Contents

Theses slides are core slides of the NWPER'2000 and IWPC'2000 talks, given by Kurt Nørmark in May and June 2000. These stand for 'Nordic Workshop on Programming Environment Research' and 'International Workshop on Program Comprehension'. There will be separate slide trails for each of these two talks (NWPER trail and IWPC trail).

Program comprehension plays a major role in any non-trivial software development effort

Two basic program comprehension approaches Prevenient The program understanding is written as a forethought before, during, or after the development of the source program Posterior The program understanding is extracted from the source program by means of special reverse engineering tools Here we want to make a distinction which represents when the program comprehension is dealt with relative to the time where it is actually needed. The prevenient approach represent the cases in which some documentation is explicitly written before it is needed. The posterior approach, on the other hand, represent the cases in which the comprehension is attempted to be extracted from the program, when it is needed typically in a late phases of the program life time.

The prevenient approach to program comprehension is hypothesized to be a very good investment seen in relation to the total life time costs

Kinds of understanding Slide Note Contents Index References

There are many different ways to classify program understanding and program documentation. Here we present a classification of program understanding which focuses on the documentation needs before, during, and after the central development efforts.

System and program understanding can be classified in many different ways We hypothesize that several kinds understanding can benefit from the ideas of elucidative programming

A classification of program-related descriptions and discussion Proactive mental understanding Design and architectural issues Oriented towards the author or co-developers Process understanding Descriptions that keep the practical and everyday activities on track Diaries and logs Maintenance understanding Conveying relevant information from the developers to maintainers Best to write after the development is finished

Plan of the IWPC talk Slide Note Contents Index References

On this page we outline the plan of the IWPC talk, which includes selected pages from this collection of material

Literate programming background An example of an elucidative program The requirements for an elucidative programming environment Model and concepts The elucidative tools Concluding remarks

Plan of the COT talk Slide Note Contents Index References

On this page we outline the plan of the COT talk in Aarhus, august 2000

Literate programming background An example of an elucidative program Model and concepts The elucidative tools Concluding remarks

Plan of the SIGDOC talk Slide Note Contents Index References

On this page we outline the plan of the SIGDOC talk, which includes selected pages from this collection of material

Background Briefly about literate programming Characteristics of Elucidative Programming An example of an elucidative Java program Model and concepts The elucidative tools Perspectives and concluding remarks

There are many kinds of documentation: analysis documentation, design documentation, interface documentation, internal program documentation, user-oriented documentation, ...

We focus on internal program documentation as a contrast to interface documentation Integrated documentation Addresses the connection between program and documentation Several possibilities will be explored

Our tools are probably useful in any situation, where there is a need to write about a program

Views on documentation Slide Note Contents Index References

There are many different views on documentation, including common attitutes that are reconfired among programmers to such a degree, that most of us belive they are true.

Many programmers are reluctant to read and write program documentation

Documentation of program understanding (DOPU) Programming requires understanding of both the problem and its solution We cannot afford to forget the understanding once established It is expensive to reestablish the understanding if its forgotten.

We see documentation as an investment in more high quality programs We hypothesize that the investment is payed back during the life time of the program We call for a professional attitude towards this problem

Integrated program documentation Slide Note Contents Index References

We will here explore the possibilities of integrating programs and written program documentation

There are several ways to integrate program and documentation

The documentation is embedded in the program Conventional program comments The program is the main structure The program is embedded in the documentation Literate Programming The documentation is the main structure The program and the documentation are separated, but connected via links Hypertext approach The starting point of Elucidative Programming

Knuth invented literate programming in order to explain and publish his programs in articles and books

Characteristics of literate programming: Detachment of program explanation structure from program structure The proximity between program fragments and their explanations The pieces of program 'live in' the documentation The opposite approach of conventional program commenting

Literate programming is typically supported by tools in the WEB-family

An example of a literate program Slide Note Contents Index References

For those who do not have a background in literate programming we here show an example of a simple literate program. The program is in Danish, but it should nevertheless be possible to get a general understanding of the approach.

In order to be concrete we briefly present an example of a literate program and the source from which it is generated

References A literate Life Program in Pascal - NuWeb - in Danish An example of the literate program source program (section 5) - in Danish

Internal structure of a literate program (1) Slide Note Contents Index References

Internally, a literate program is made by a hierarchical program and a linear document structure

Figure. The internal literate programming structure.

Internal structure of a literate program (2) Slide Note Contents Index References

The tangle tool extracts and assembles the source program

Figure. The program as assembled from the WEB above.

The WEB tools for literate programming Slide Note Contents Index References

There are two classical WEB tools for literate programming. We here show the roles of these two tools.

There are two central WEB tools: weave and tangle

Figure. The 'classical illustration' of the processing done by the WEB tools tangle and weave.

Problems with WEB-like literate programming Slide Note Contents Index References

Elucidative programming is meant to be an alternative to WEB-like literate programming. Therefore it is natural to identify a number of problems with classical literate programming which we attempt to remedy in our variation of literate programming.

The Program is embedded in the documentation Only few programmers want to split their programs in fragments and to organize these as parts of the documentation Language mixing The mix of three languages in a single document is not attractive Documentation language, programming language, WEB interconnection language Reading and comprehension problems in the WEB-source files Extra level in error handling Error handling is more complicated because the errors need to be mirrored in the WEB source program Reasonable solutions exist Paper orientation Most existing tools are oriented towards printed paper presentation of the literate program

Despite these problems the original ideas and tools remain to be very interesting and attractive for skilled and motivated users

To elucidate is

"to make clear or plain, especially by explanation"

"to throw light on something complex"

Elucidative Programming Slide Note Contents Index References

At this place we introduce elucidative programming. We here describe a number of key properties of the documentation approach.

Elucidative programming is meant to be a variant of literate programming for prevenient program comprehension which remedies some of the identified problems with literate programming

Elucidative programming highlights: User interface Present the program and the documentation side-by-side in an Internet browser Leave the program "in peace" Does not impose any requirement on the organization of the program in files or directories Address existing abstractions from the documentation Does not rely on separately named program fragments Supported by the development environment Requires specialized editor functionality We will return to all of these items later in the presentation

Requirements Slide Note Contents Index References

The first paper we wrote about elucidative programming was focussed on a number of requirements. These requirements are listed on this page.

Target The internal documentation must be oriented towards current and future developers of the program Purpose The internal documentation is intended to address explanation which serves to maintain the program understanding and to clarify the thoughts behind the program Separation The program source file must be intact, without embedded or surrounding documentation Tools The programmer must experience support of the program explanation task in the program editing tool Chunking The program 'chunking structure' follows the main abstractions supported by the programming language Medium The documented program must be available in an attractive, on-line representation suitable for exposition in an Internet browser

An example of an elucidative program Slide Note Contents Index References

In order to be concrete we will here give an example of a literate program. The program is referred in the NWPER'2000 paper, which is being presented during these slides.

The example illustrates the development of a small program that converts a large integer, representing the number of seconds elapsed since January 1, 1970 to a conventional date and time structure

References The Time Conversion example An everyday example: The annotator tool

An example of an elucidative program Slide Note Contents Index References

In order to be concrete we will here give an example of an elucidative program. This program is akin to the program which is discussed in the SIGDOC paper

The example illustrates the development of a small program that converts a large integer, representing the number of seconds elapsed since January 1, 1970 to a conventional date and time structure

Internal comment

- Her kommer Java eksemplet fra Max

References The Time Conversion example (Java - Only in Netscape) The Time Conversion example (Scheme) An everyday example: The annotator tool (Scheme)

Examples of elucidative programs Slide Note Contents Index References

The examples illustrate a demo program in Scheme, a real life LAML tutorial based on Scheme , and two educational Java programs

Elucidative Scheme Programs The Time Conversion example The LAML Tutorial Elucidative Java Programs Java MIP example 1 Java MIP example 2

The elucidative programming model Slide Note Contents Index References

This page describes a number of important concepts behind the elucidative programming model.

Entities Named abstractions on the program side Sections and subsections on the documentation side A naming scheme that allows us to address the program entities from the documentation Relations A doc-prog relation which connects explanations with program entities. A prog-prog relation which connects applied and defined name occurrences A doc-doc relation which connects explanations across the hierarchical documentation structure Source markers Represent positions in a program which we want to address in the explanations Documentation bundle An aggregation of programs, documentation texts, and additional properties (e.g., processing options)

The source format of the documentation - Scheme Slide Note Contents Index References

When the programmer works with a literate program the source format of the documentation is of particular interest. This format is important because it constitute the words and lines actually written by the programmer. In addition, the programmer will spend much time reading parts of the documentation in this format. Therefore the properties of the format matters. This page is related to the Scheme elucidator only.

A simple, specialized markup is used for the overall structure, the relations, and the source markers HTML is used for additional markup purposes

Observations about the source format of the documentation Non-obstructive, discrete, and terse at least compared with XML-like markup It is relatively easy and attractive to read the documentation in the editor The supplementary use of HTML is in line with the choice in JavaDoc Avoids proliferation of languages with similar purposes The mixing of markup styles is not aesthetically pleasant

References An excerpt of the time conversion documentation source The Time Conversion example

The source format of the documentation - Java Slide Note Contents Index References

When the programmer works with a literate program the source format of the documentation is of particular interest. This format is important because it constitute the words and lines actually written by the programmer. In addition, the programmer will spend much time reading parts of the documentation in this format. Therefore the properties of the format matters. This page is related to the Scheme elucidator only.

An XML markup language is used for the overall structure, the relations, and the source markers HTML is used for additional markup purposes

Tool Overview Slide Note Contents Index References

There are three important tools in an elucidative programming environment. Of these, a standard Internet browser counts for one. The others are the editor used for programming and documentation, and the tool which produces the underlying browser format.

Our elucidative programming environment contains the following tools: An elucidator which converts programs and documentation to HTML pages An editor which is used for both programming and documentation purposes A browser which is used for presentation of the elucidative program

The elucidator plays the role of the literate programming weave tool In elucidative programming there is no need for a tangle tool

The elucidator tool Slide Note Contents Index References

The elucidator tool produces the HTML pages to be shown in a browser. In addition, the elucidator tool makes a number of data structures used by the editor to gain knowledge about the documentation bundle.

The elucidator is the tool that processes a documentation bundle in order to produce an elucidative program

Tasks of an elucidator Abstraction Parses the documentation and the program Extracts knowledge about program and documentation entities Synthesis Generates the HTML pages with links that represent the three kinds of relations between entities

The synthesis part of the processing may be carried out by the WWW server on a dynamic basis

The editor tool Slide Note Contents Index References

The editor tool for the elucidator plays a major practical role. It is important that the programmer can continue working in his favorite environment. We will assume that our programmers use Emacs. Therefore we chose be augment the Emacs text editor with elucidator support.

It is of major importance to provide for easy and flexible editor support of elucidative programming We have augmented Emacs with elucidative facilities

Design objectives: The editor applies a split-screen layout in the same way as the browser The editor's support of the programming language is available in non-changed form The creation of the doc-prog and doc-doc relations is nursed in particular The major navigation possibilities - as supported by the browser - are also present in the editor Emacs keeps the documentation bundle together by a special marking of the buffers involved

The browser tool Slide Note Contents Index References

The browser tool of an elucidative programming environment is just a standard, modern Internet browser. On this page we emphasize the typical frame layout used by an 'elucidative browser'.

The browser tool is realized by an ordinary Internet browser

Characteristics of the browser The "T layout" of the browser window, supporting a side-by-side alignment of documentation and program

Figure. The basic frame layout of an Internet browser that presents an elucidative program

Reference The Time Conversion example

Tool Integration Slide Note Contents Index References

Tool integration is a key issue when two or more program development tools are used together. On this page we discuss integration issues of the tools in an elucidative programming environment.

The current integration of the editor and the elucidator is simple and rather primitive

Cooperation between the elucidator and the editor The elucidator abstracts information from the documentation bundle which the editor can read on demand The editor is always a step behind with respect to knowledge about the current documentation bundle The elucidator can be activated very smoothly from the editor

Mutual activation of the browser from the editor - and vise versa - would be a major practical improvement

Tool philosophy Slide Note Contents Index References

Here we will discuss possible and different tool organizations. We call it 'tool philosophy'. Our point is not to force the programmer into a specialized environment for authoring of program documentation. We go for documentation enabling of existing environments. In our experience, programmers are rather conservative, and most of them want to continue working in their favorite environments. There must be major advantages in a new environment in order for the programmers to shift.

Possible tool organizations: Provide a brand new tool with an embedded editing environment for elucidative programming Like Samtinger's Dogma System Introduce a separate tool and augment an existing editor with knowledge about elucidative programming Typically a Unix and Emacs solution Include elucidative aspects into current integrated programming environments Long term goal - requires industrial collaboration

Programmers do not want to be forced into a new and specialized editing environment

The Scheme Elucidator Slide Note Contents Index References

On this page we will address the fact that the concrete tool supports the programming language Scheme. Scheme is dialect of Lisp, and as such, Scheme is language with prefix and parenthesized notation. There is also an Elucidator for Java.

The elucidator and the editor tools are both language dependent

Observations about the Scheme elucidator: Basically, we address Scheme define forms from the documentation side Not arbitrary program fragments as in WEB-like literate programming In Scheme it is possible to have program elements outside abstractions Only relevant for imperative programming How are these addressed from the documentation? The solution: Sectional comments which names the form(s) following the comment Lexical comments cause problems in the abstraction process Not only in Scheme, but almost in every programming language We apply a preprocessor which transform all lexical comments to syntactical comments

Scheme was chosen because it has been my main language for Internet related programming purposes

Reference The Time Conversion example

The Java Elucidator Slide Note Contents Index References

The Java Elucidator has been developed by five master students The first version is similar to the Scheme elucidator at the conceptual level The second version supports a more structured documentation approach

Observations about the Java elucidator: The naming scheme used to address program units from the documentation is rather complex Much more elaborate to make a doc-prog link Relies on an abstraction process which populates a relational database with program and documentation information The synthesis task is done dynamically by the WWW server It is also possible to generate a static set of HTML pages

The Design of the Java Elucidator Slide Note Contents Index References

On this page we will illustrate the overall design of the Java Elucdiator.

Figure. The design of the Java Elucidator

Minor features Slide Note Contents Index References

On this page we will list of number of minor details in the elucidative tools. Take alone, non of these features are of great importance, but together they may perhaps make a difference.

There is a number of finer details of the elucidator tools that are worth mentioning

The use of tooltips-like pop-up texts gives useful additional information from behind the scene The cross reference index can be accessed conveniently from any program entity Provides for flexible traversal of selected calling chains Access to an on-line programming language report from the browser that presents the elucidative program Editor features: A command to create a new documentation bundle: M-x make-elucidator A command to bring up all parts of an existing documentation-bundle: M-x setup-elucidator Activation of the elucidator tool from the editor: M-x elucidate

NWPER'2000 Questions Slide Note Contents Index References

Here are my answers to the questions that where asked to all NWPER'2000 speakers.

What is the main novelty/contribution of the paper? A WWW approach to internal program documentation A variation of literate programming which 'leaves the program in peace' How will this novelty/contribution improve SD practice or SD research? It could perhaps demonstrate an attractive way to document program understanding An educational effort will be needed to make a practical impact What are the main problems with the novelty/contribution and/or with the paper? The general reluctance to document any understanding related to the program as such The problem of what to document when Can the proposed approach be expected to scale to real-life problems? Probably not in its current elaboration We are working on variations which we hope will

A broader perspective on the use of an elucidator Slide Note Contents Index References

Here we will list a number of potential areas which we expect can be supported by the elucidator.

Most documents that need to address program units or program details will be able to benefit from the ideas of elucidative programming

Documents with 'an elucidative potential': Diaries and logs Written code reviews and code walk throughs Program tutorials Design documents Maintenance documentation

The Java Elucidator has been tried out by a group of Dat1 students in the fall of 2000

We have examined the usefulness of Elucidative Programming in seven different areas: Review situations Confidence of knowledge about software Further confidence through inclusion of tests Maintenance of Elucidative Programs is manageable during development Programs become less error prone Ease of maintenance Generally, the tool is useful

Industrial experience Slide Note Contents Index References

The Java Elucidator has been used by three programmers in Danish Java software house. This experiment was carried out by Max R. Andersen and Claus N. Christensen

Feedback: Elucidative Programmig is applicable in industry The tools were addapted - but they probably need improvements Problem with two representations (development, browser). Abstraction requires that the program can be compiled A group communication potential was revealed Not easy to capture and represent verbal communication

Industrial quotes Slide Note Contents Index References

"Elucidative programming is a paradigm which fits the lightweight process we use" "I believe that if we did not have such an Elucidator tool, then in the best case this would be scribbled down on a piece of paper, which would eventually be scrumpled up " "If I was to write the documentation completely isolated from the source code - well then it would have been completely useless - so it has to be done on the source code in some way" "At first it feels like: this takes time. But I do not think this is necessarily the case" "This is not end user documentation, and it is not polished analysis and design. It is actually very intimate, so on that set it reminds me a lot of this diary I have written from time to time" "It is probably something you has to integrate in the process... to get it on your mind, as a daily activitiy, as a natural part, right?" "I do not dare to mess with this, but I can always make a note about it - if I one day want to change something"

Contributions: A renewal of the tool ideas for literate programming A serious attempt to present programs and their internal understanding on the Internet Status A complete and operational prototype of an elucidative Scheme environment exists A experimental and operational prototype of an elucidative Java environment environment exists

The Java Elucidator is used and investigated by Thomas Vestdam in his Ph.D. project

References elucidator.sourceforge.net http://www.cs.auc.dk/~normark/laml/

The Elucidative Programming Home Page http://www.cs.auc.dk/~normark/elucidative-programming/ The DOPU Home Page Software Quality through Documentation of Program Understanding http://dopu.cs.auc.dk

These slides are available from the elucidative programming home page Both the Scheme and Java Elucidators are pieces of free software

Elucidative Programming
Course home     Author home     About producing this web     Previous lecture (top)     Next lecture (top)     Previous lecture (bund)     Next lecture (bund)     
Generated: December 2, 2005, 13:12:37