Kairos Laetus-in-Praesens: Kairos

Indications

Technical considerations

 


How Kairos works: a technical summary by Claude AI

Source pages (PHP/HTML, ~2,500 documents, 1960s–2020s) | read in "scraping mode" v OpenInsight processing pipeline (16 procedures, AA01–AA16) | writes plain-text files v 15 semicolon-delimited CSV files, one per record type, one shared 24-field layout | imported by a custom Drupal module v Drupal 11 display (about 50,000 records, Views, search, statistics)
    1. Source: The documents are PHP pages arranged in decade folders (docs60s … docs20s and others), each carrying standard metadata (title, author, date, keywords, description) and a body divided by section headings. Images, animations and 3D (X3D) models sit in folders beside the pages and are never copied into the database: Kairos records only point to them.
    2. Processing (OpenInsight): The processing is done in OpenInsight, a MultiValue database with its own BASIC+ language, in which the author has maintained the indexing of his work for over twenty years. Sixteen procedures run in sequence under a single driver:
Stage What it does
AA01–AA02 Read every source page; split it into its sections; extract titles, years and metadata.
AA03 Resolve every link in every page: links between documents become stable Kairos addresses, and are counted as citations in both directions.
AA04–AA05 Tables of contents; the document records themselves (introductions and sections).
AA06–AA10 Bibliographic references, authors cited, cross-reference records, duplicate detection.
AA11–AA12, AA16 Visual records: images, animations, galleries, and tables (with and without images).
AA13–AA14 Indexing; clustering of documents under 54 topics by keyword.
AA15 The statistical summary (documents, citations and topics by decade), generated as a ready-made page.

The output is fifteen plain-text CSV files. Every record, whatever its type, has the same 24 fields (type, address, titles, year, citation counts, body and so on); the meaning of some fields varies by type, and is documented in the procedures and in the column labels of the Kairos listings. Each record's address (its "alias") is built from the source file's name plus a suffix for its type and section, so addresses are stable from one regeneration to the next.

A good deal of the pipeline is defensive. The source pages span six decades of authoring habits and tools, so the procedures repair character-encoding damage, escape quotation marks, and check that every record has exactly the right number of fields before it is written. They also keep censuses of anomalies (undated pages, conflicting titles, broken links), which become worklists for correction at source, never in the display.

  1. Display (Drupal 11): The CSV files are uploaded to the web server and imported into Drupal 11 by a small custom module (migrate_helios), built on Drupal's standard migration framework. All records go into a single content type with 23 fields; each import matches records by their address, so a re-import updates what exists, adds what is new, and lists (optionally removes) what has disappeared from the source. A command-line driver imports all fifteen files in one unattended run, checking their encoding first and logging the outcome per type.

    Drupal then provides what a set of static pages cannot: the filterable listings (Documents, Authors cited, References, Visuals), full-text search, and stable addresses for every section of every document. A minimal custom theme reproduces the plain appearance of the earlier site and adapts the layout to phone screens. The explanatory Directions pages are deliberately kept as static HTML files outside Drupal

Scale (October 2026)

Documents (introductions) about 2,500
Document sections about 13,600
Citation links between documents about 39,800
External works cited / authors cited about 30,500 / 12,900
Images / animations / tables / galleries about 6,500 / 4,800 / 4,500 / 640
Topics 54, in 14 groups
Records in the display layer about 50,000

Future-proofing: what has been designed to last

Acknowledged weaknesses

Conversion by program: As described in more detail in the Content overview, the documents on this Kairos site derive from a set of articles held in PHP format on the related Laetus-in-Praesens website. These typically longer "articles" have been split into the shorter "documents" imported into the free and open-source content management system Drupal.

The approach taken to the conversion of PHP articles into Drupal records is one which reflects the biases of a programmer exploring solutions such as to avoid manual copying of portions of those articles into the content management system (CMS). The challenge was whether the conversion could be extensively ensured by program, notably in order to extract other information and build that into other records to be imported into the CMS. The intention was therefore to build many of the CMS records prior to importing rather than endeavour to generate additional content types within the CMS facility. The approach was framed in this way because of extensive expertise in manipulation of text with a DOS-based application -- and little expertise in the PHP-related programming required for the CMS.

The approach was successively enabled using a suite of programs developed with the Advanced Reverlation (AREV) application through a DOS box in Windows XP. The resulting pre-formatted flatfiles (CSVs) were then imported into Drupal 6 and later Drupal 7. This required some ingenuity to circumvent a 64k constraint on record length within AREV for documents of greater size. Cessation of support for Windows XP, encouraged a shift to Open Insight, a later variant of AREV. The upgrade to the Kairos site in December 2018 was enabled using Open Insight 9.4. Considerations is being given to upgrading that facility to Open Insight 10.04.

Article formatting: Advantage has been taken of the fact that the PHP articles were in a format which had remained standard and relatively stable over decades, both from the earliest (in the 1960s), and since first placed onto a website (in HTML format) in the early 1990s. Articles from earlier periods were adapted to that format as they were digitized. The key factor enabling conversion was the presence in those files of HTML title delimiters defining the sub-titles of what could then be split out as Drupal records. The conversion challenge was defined such as to avoid any additional mark-up -- otherwise required to facilitate the process, using programming "tricks" to circumvent anomalies. This could well be described as a less than efficient process (if not stupid!), but it did offer some nice programming challenges for someone anxious to avoid manual manipulation (at all costs!).

Retaining relationship to original version: The conversion challenge was also seen as a means of preserving a degree of complementarity between the PHP articles in the Laetus facility and the variant on the Kairos facility. The intention was not to switch to writing articles within the Drupal CMS, or updating them there, since it has been far more convenient to continue the process of writing/editing of the PHP variants within the Laetus facility using Dreamweaver. This was one reason for using record (node) identifiers within the CMS based on the original PHP file/folder name -- rather than switch to a numeric node identifier as is most commonly the case for a CMS.

Note that it is the Laetus version which is considered to be the master copy. Only minor editing is done exceptionally on documents in the Drupal context -- most notably in the event of conversion issues which have not been resolved by program. Errors emerging in the Drupal may well be used as a means of detecting and correcting errors in the Laetus version or in the conversion scripts.

Constructing records for import: Building the various CMS record types prior to import, rather than depending on (absent) Drupal skills to manipulate the basic imported documents, has meant that new record types can be created and populated as required in order to enhance the CMS facility. Of particular interest are those relating to the pattern of links.

Drupal node import: Of interest in the strategy adopted is the constraint imposed by the state of development of the Drupal "node import facility". Basically the options available for updating any node of a particular content type by the import process are to delete such nodes individually, or in a batch process (VBO) -- and then to import the corrected set of nodes into the IDs thereby made "free". The provisions for "overwriting" a node, without prior deletion, have been progressively developed stage within the Drupal community -- but primarily for numeric nodes. In practice this means that it is easy to batch delete all the bibliographic reference records, or the associated author records, and then to re-import a set generated from PHP articles after corrections (in the light of errors that became apparent from Drupal sorts). The advantage of the alphanumeric node naming system is that the links to the other documents are not affected by this process since the pointers from those documents remain valid.

Upgrading and adaptation:

Character encoding issues: Working with articles published as early as the 1960s, making extensive use of accented characters, has required a degree of flexibility in adapting the conversion to handle characters which pre-date the currently favoured UTF-8 standard. Some of the articles are in French and other languages. Many cite authors and articles requiring such characters. Although the conversion process enabled some of these anomalies to be "corrected", These issues have not been completely resolved and are reflected in the contrating ways in which Drupal itslef handles such encoding in different contexts.

Augmenting access possibilities: A major motivation for exploring a CMS variant was to segment the longer PHP files (some over 150k) into more "readable" forms as CMS records. This was seen as particularly valuable in that the sub-titles attributed to the HTML title-delimited segments were interesting to extract in order to benefit from the Drupal Views facility, in addition to enabling more speciic access via search engines.

Benefitting from extensive hyperlinking: A significant characteristic of the PHP articles is the degree of hyperlinking between them. The conversion was designed to derive further information from this pattern of links, notably by generating "checklists" of citations "from" and "to" the CMS records. Unfortunately, as noted above, the links have been enabled to the "main" document introducing a set, and not to the individual documents of the set. This could be improved in the future.