Parthenon frieze at the Filozofski fakultet Sveučilišta u Zagrebu

profilecroatianneolatin.blogspot.com

Facultas philosophica Universitatis Zagrabiensis

The individual object in Arachne

The individual object in Arachne

Naredbom bogoštovno-nastavnoga odjela kr. zemaljske vlade, br. 7816 od 9. srpnja 1892. nabavljene su od tvrdke formatora i modelara muzeja D. Brucciani et Comp. u Londonu za arheološki odjel narodnoga muzeja sadrene kopije Parthenonovih skulptura (Elgin marbles) koje se nalaze u britanskome muzeju. Kopije, koje su stajale do 8000 for., privremeno su smještene u jednoj dvorani obrtne škole.
Muzej sadrenih otisaka u Zagrebu. // Viestnik Hrvatskoga arheološkoga društva, n.s. I (1895). Zagreb: 1895-1896., str. 114.

Martina Matijaško. 2012. "Gipsani odljevi antičkih umjetničkih djela smješteni u prostorijama Filozofskog fakulteta u Zagrebu". Muzeologija, No.46.

Thucydides in the Perseus Catalog

GHLH korpus

CroALa + CroALaBib in BaseX

XQuery for VIAF id numbers

Our bibliography has (too many) authors for whom we need VIAF numbers. Using VIAF API (as described), we get numbers from the database in several steps.

1. find distinct values of unassigned authors in the bibliography
2. turn these values into a XML sequence
3. use the sequence to query VIAF

For the last phase, we use this XQuery (and BaseX GUI):

declare namespace ns2="http://viaf.org/viaf/terms#";
declare namespace ns3="http://viaf.org/viaf/terms#";
(: address to which we are sending the queries :)
let $url := 
("http://www.viaf.org/viaf/search?query=local.personalNames+all+%22REPLACE_URN%22&maximumRecords=1&sortKeys=holdingscount&httpAccept=text/xml")
(: our sequence :)
let $rijeci := 
<a>
  <n>Adelmann von Adelmannsfelden, Konrad</n>
  <n>Aegidii, Guillermus</n>
  <n>Alexander, Natalis</n>
  <n>Algerus</n>
<!-- many more -->
  <n>Zoller, Martin</n>
</a>

(: for each item in sequence, batches of 100 :)
for $r in $rijeci/n[401>position() and position()>=301]

let $qrijeci := replace($r, " ", "+")
let $parsed := (doc(replace($url,'REPLACE_URN',$qrijeci)))

return element author { 
element ref { 
attribute type { "viaf" } , 
attribute target { data($parsed//ns2:VIAFCluster/ns2:viafID) } , 
data($r) } 
}

Sound of music

A testimony of Stefano Fieschi da Soncino on young singers from Dubrovnik in August 1441, entered in a somewhat inappropriate place: in the official notebook Diversa Cancellarie. Published by Konstantin Jireček in 1897.

... Darüber hat sich ein merkwürdiges Zeugniss erhalten, kalligraphisch eingetragen im Buche »Diversa Cancellarie« 1440 f. 158, unter der Zeichnung einer Krone, zum 19. August 1441:

Commemoratio suauitatis cantus dominorum camerariorum. Cum ego Stefanus Flischus Soncinensis, cancellarius Raguseus, ultrascriptam dominorum camerariorum pacti et sortis conuentionem describerem (d. h.: in eo tempore vendemiarum, in quo unus saltem ipsorum qualibet die se debet Magnifico domino Rectori presentare, soll jeder es für 10 Tage thun), tunc ipsi Ser Nicolaus Pauli de Goze, Ser Marinus Junii de Cruce atque Ser Volcius Blasii de Babalio ita me coram suauissime cecinerunt, ut mihi audire visum fuit amenissimam quandam celestem armoniam. Ego vero, qui tum scribendo, tum etiam tota die complures libros peruoluendo aliquantisper defessus fueram, maxima profecto illius amenissimi cantus suauitate oblectatus sum. Et quamquam ipsi domini camerarii hoc mihi inuere videbantur, quod ipsi, magna eorum in me beniuolentia commoti, illam tam diuinam armonie suauitatem me coram egissent, illud tamen non preterit, quin ipsi venustissima egregiarum amicarum suarum forma compulsi et suauium ipsarum morum diligentissime memores, tunc temporis tam diuinitus cecinerunt. Magnas tamen ipsis gratias ago, qui tam suauissima amenitate me oblectarint, sed maiores habeo illis prestantibus eorum amasiis, que ipsos ad illos cantus impulerunt. Valeant ergo insignes et pulcerrime domicelle, quarum amor, mores et nobilitas tantam vim habent, ut tam prestantium iuuenum mentes ad se alicere valuerunt et eorum voluntates, in quancunque partem velint, faciliter impellere possunt! Valeant etiam ipsi domini camerarii, qui tametsi magna dignitatis auctoritate prediti sunt, non tamen tanta dulcedine ullo modo me carere passi sunt ! Valeat denique hec magnifica atque florentissima ciuitas Ragusea, que iuuenes tam insignes tamque prestantes procreauit, qui splendidissimum sue rei publice decus et ornamentum existunt! Qui cum etate maturiori creuerint, tunc huius alme ciuitatis statum non solum illesum optime conseruabunt, verum etiam acuratissima eorum prudentia mirandum in modum amplificabunt, ad quam comoditatis gratiam utinam illos deus preseruare dignetur. Ex cancellaria celeberrime urbis Ragusee 14 Kal. Septembres, tunc celorum constellationibus dulcissimam eorum amenitatem in ipsos dominos camerarios diuinitus influentibus.

An SQLite SQL for homonyms

In an SQLite database there is a table of parsed (Latin) word-forms, like this:

tokenID|token|code|lemma|type
1266165|rutilantis|v--pppafa-|rutilo||newmorph
1266166|rutilantis|v--pppama-|rutilo||newmorph
1266167|rutilantis|v--sppafg-|rutilo||newmorph
1266168|rutilantis|v--sppamg-|rutilo||newmorph
1266169|rutilantis|v--sppang-|rutilo||newmorph

We are interested in cases where contents of the token field are the same, but code is different. Code holds grammatical information; the first letter is a shorthand for part of speech (v = verb in the example above). For the moment, we can retrieve this for a specific word, using the following SQL query:

select distinct code1 from (
   select substr(code, 1, 1) as code1 from (
     select code from Lexicon where token like "verum")
      );

Grouping on two fields (a recipe found on Stack Overflow) seems promising:

select token , code, count(*) 
 from Lexicon 
 group by token collate nocase, code 
 having (count(*)>1);

And I think this would be the final query:

select distinct token , c 
from (select token , substr(code, 1, 1) as c 
      from Lexicon 
      group by token collate nocase, code 
      having (count(*)>1) limit 30);

Now off to check it on all 1,257,854 rows in the Lexicon table.

It took a bit of post-processing with a bash command (had to swap POS and Wordform fields):

sort hom.csv -t, -k1 \
| sed 's/\([^,]*\),\(.\)/\2,\1/g' - \
| uniq -D -s 2 > hom-pos.csv

The results are now publicly available as a Google Fusion table, all 19,350 rows of them: homonyms (and homographs) differing by part of speech, found in a real digital corpus of Latin texts.

Finding homonyms in a (Latin) treebank

Homonyms and homographs (H & H) in a language are a good thing to master — a lot of confusion goes away once we have understood the differences, and our grasp of the language is significantly improved.

We don't want to run away from the H & H — we want to tackle them at full speed. That means that, if we read a text, we could pick from it a list of phrases with H & H, read them, and see which meaning is employed where.

To do this, we need:

• a list of homonyms and homographs, such as the one Chris Francese and David Kovacs have compiled for Latin
• a text
• tools to find phrases with H & H and collect them in one place

Our task can be done in many ways, including purely "manual" ones. But we would like to use the Perseus Latin Treebank files to find more homonyms and homographs, and to extract phrases from these treebank files as well.

Finding H & H turns out to be a computationally demanding task. On some 50,000+ words my computer chokes and never finishes; 10,000 words is still too much for it. But sets of some 2,500 words are about right, don't take all night to finish.

To an XML file like this:

<uniq>
<w form="vulgo" postag="d--------"/>
<w form="vulgo" postag="d--------"/>
<w form="vulgus" postag="n-s---mn-"/>
<w form="vulnera" postag="n-p---na-"/>
<w form="vulnera" postag="n-p---na-"/>
<w form="vulnera" postag="n-p---na-"/>
<w form="vulnerat" postag="v3spia---"/>
<w form="vulneratum" postag="t-srppma-"/>
<w form="vulnere" postag="n-s---nb-"/>
<w form="vulneribus" postag="n-p---nb-"/>
<w form="vulneribus" postag="n-p---nb-"/>
<w form="vulnus" postag="n-s---nn-"/>
<w form="vulpes" postag="n-p---fn-"/>
<w form="vult" postag="v3spia---"/>
<w form="vult" postag="v3spia---"/>
<w form="vult" postag="v3spia---"/>
<w form="vult" postag="v3spia---"/>
<w form="vultis" postag="v2ppia---"/>
<w form="vultu" postag="n-s---mb-"/>
<w form="vultu" postag="n-s---mb-"/>
<w form="vultum" postag="n-s---ma-"/>
<w form="vultum" postag="n-s---ma-"/>
<w form="vultum" postag="n-s---ma-"/>
<w form="vultus" postag="n-p---ma-"/>
<w form="vultus" postag="n-p---ma-"/>
<w form="vultus" postag="n-p---ma-"/>
<w form="vultus" postag="n-p---ma-"/>
<w form="vultus" postag="n-s---mg-"/>
<w form="vultus" postag="n-s---mn-"/>
</uniq>

We apply the following XQuery (using BaseX, in my case):

element uniq
{ let $a := //*:w
for $l in distinct-values($a/@form),
$f in distinct-values($a[@form=$l]/@postag)
return

element w {
attribute form { $l },
attribute postag { $f }
}
}

Result:

<uniq>
<w form="vulgo" postag="d--------"/>
<w form="vulgus" postag="n-s---mn-"/>
<w form="vulnera" postag="n-p---na-"/>
<w form="vulnerat" postag="v3spia---"/>
<w form="vulneratum" postag="t-srppma-"/>
<w form="vulnere" postag="n-s---nb-"/>
<w form="vulneribus" postag="n-p---nb-"/>
<w form="vulnus" postag="n-s---nn-"/>
<w form="vulpes" postag="n-p---fn-"/>
<w form="vult" postag="v3spia---"/>
<w form="vultis" postag="v2ppia---"/>
<w form="vultu" postag="n-s---mb-"/>
<w form="vultum" postag="n-s---ma-"/>
<w form="vultus" postag="n-p---ma-"/>
<w form="vultus" postag="n-s---mg-"/>
<w form="vultus" postag="n-s---mn-"/>
</uniq>

The most interesting cases are those in which @postag attribute begins with a different value for the same @form, e. g:

<w form="vivis" postag="a-p---mb-"/>
<w form="vivis" postag="a-p---md-"/>
<w form="vivis" postag="n-p---mb-"/>
<w form="vivis" postag="v2spia---"/>

Then we look for "vivis" e. g. in the Croatiae auctores Latini text collection.

How to write a BaseX XQuery with RESTXQ

Caution: technical stuff. Over holidays we managed to put up a BaseX XML database instance as a web application on several machines. But how to execute an XQuery there? A simple approach: use an already provided BaseX REST interface. However, the BaseX team seems more interested in RESTXQ, "a set of XQuery 3.0 Annotations and a small set of functions to enable XQuery to provide RESTful services, thus enabling Web Application development in XQuery" (from the unofficial RESTXQ draft). BaseX supports RESTXQ very well, but the existing documentation is somewhat sparse for a non-programmer like me. Conspicuously absent is an example of a "standard" XQuery search directed at a database (or, in XQuery parlance, a collection). This will be provided here.

The task. A BaseX war instance is deployed on a Jetty server (on my machine, which runs Debian Mint, in /var/lib/jetty8/webapps), accessible on the address http://localhost:8080/BaseX772. A database collection crobib was created and populated with several TEI XML files (with FRBR-structured bibliographical data on Croatian Latin authors, works, and manifestations). We want to execute the following query over the internet, finding the text under tei:persName element as child of all eleventh tei:person elements in the collection:

declare namespace tei = "http://www.tei-c.org/ns/1.0";
for $i in collection("crobib")//tei:person[11]
return element p { $i/tei:persName//text() }


The solution. An .xq script should be written and placed (in our case) under the root of the BaseX war archive. If all goes well, it is found and read by Jetty and BaseX when the server is restarted. This is the script (cbxq.xq). Note how the resulting sequence has to be wrapped in a div element:

import module namespace rest = "http://exquery.org/ns/restxq";
declare namespace page = 'http://basex.org/examples/web-page';
declare namespace tei = "http://www.tei-c.org/ns/1.0";
declare %rest:GET %rest:POST %rest:path("person")
function page:person() {
element div {
for $i in collection("crobib")//tei:person[11]
return ( element p { $i/tei:persName//text() } )
}
};
return

The script is requested over the following address: http://localhost:8080/BaseX772/person.

Going further. We want to search not just for eleventh tei:person element, but for whichever we want. The number of the element should be turned into a variable holding an integer, and the variable will be given as part of the HTML address request. The script now looks like this:

import module namespace rest = "http://exquery.org/ns/restxq";
declare namespace page = "http://basex.org/examples/web-page";
declare namespace tei = "http://www.tei-c.org/ns/1.0";
declare %rest:GET %rest:POST %rest:path("person")
%rest:query-param("var", "{$var}")
function page:person($var as xs:integer) {
element div {
for $i in collection("crobib")//tei:person[$var]
return ( element p { $i/tei:persName//text() } )
}
};
return

We had to declare query parameter var: %rest:query-param("var", "{$var}") and to instruct the function page:person to expect it: function page:person($var as xs:integer).

The script is requested with a call such as this (querying the hundredth tei:person): http://localhost:8080/BaseX772/person?var=100.

Ancient Greek as a Unicode Character Class

In oXygen XML editor, when we want to search for any Greek character (using Perl character classes), we do it with this regular expression:

\p{IsGreek}

Simple, great — and took me half an hour to find. Is this programming or philology?

Elephas culicem non curat

Phalaridis epistula LXXXVI, Graece ed. Hercher 1873 apud Didot (et Heml Lace):

πσ. Ἱέρωνι. Πόλλα λέγειν ἔχων καὶ κατὰ σοῦ καὶ περὶ ἧς κατ' ἐμοῦ πεφλυάρηκας ἐν Λεοντίνοις δημοκοπίας οὐδὲν ἐρῶ περισσότερον πλὴν ὅτι κώνωπος ἐλέφας Ἰνδὸς οὐκ ἀλεγίζει.

Latine vertit Aretinus, Venetiis 1492/1500? (apud BSB Digital):

Hieroni. Qvom multa de te: et de concione quam contra me ad Leontinos stulte habuisti dicere possim: nolo tamen superfluis uti uerbis: nisi quod culicem elephas Indus non curat.

Ita autem vertit Joannes Daniel van Lennep, Groningae 1777 (apud archive.org, p. 119):

XXIX. HIERONI. Cum multa habeam dicere et in te, et de concionibus, quas effutiuisti apud Leontinos, discordi plenas popularitate, nihil dicam amplius, nisi elephantem Indicum non curare culicem.

Querying CTS edition of Osmanides through Fuseki

Once we have a CTS instance containing an XML edition, e. g. of Vlaho Getaldić's Osmanides, up and running, we query it through the Fuseki server, by SPARQL queries such as this one:

select ?x where {?x ?v """Fortis commisit, victorque evasit ab hoste. 165"""}

This means: find a (URN) citation for the line containing this text.

The result is:

<urn:cts:croALa:croalaget003.croalaget001.izdleipzig:165>

This was a working query. Now, here is a meaningful query:

select ?s ?p ?o where { <urn:cts:croALa:croalaget003.croalaget001.izdleipzig:10.249> ?p ?o .}

Meaning, more or less: return RDF predicate and object where the RDF subject contains value as named. This would be book 10, line 249 of the Osmanides.

A useful introduction to SPARQL can also be found under Beginner's guide to RDF: 6. Querying with SPARQL. For example, on using queries (here, with CTS namespace):

PREFIX cts: <http://www.homermultitext.org/cts/rdf/> select ?s ?p ?o where { <urn:cts:croALa:croalaget003.croalaget001.izdleipzig:10.250> cts:hasTextContent ?o .}

Text and its links

One of the things that can be done with a TEI XML texts is transforming it into other formats. E. g. into HTML. (Other thing that can be done is including the text in a collection like CroALa, of course.)

A HTML edition can have links of its own. These links can be encoded in XML, so that they "come alive" after the HTML transformation. What is needed is a good idea what to link to.

There are two natural locations. If a text is a transcription of a source, such as a manuscript, and if the source is present on the internet, we can link to page images. And, if text contains quotations of or allusions to other texts (and the texts are present on the internet), we can link to these texts — or hypotexts, as Gerard Genette would call it.

This isn't quite as simple as it seems. What if hypotext is the Bible, with many books, chapters and verses, and our text refers to a precise location in it? Over the centuries, philology developed special techniques just for such referring actions, and the techniques are migrating to the internet. Slowly, though; perhaps not surprisingly, the Bible — with services such as bib.ly — is again first to apply them.

Linking to images and linking to sources are features of our working editions of Andrija Dudić's (Andreas Dudithius, 1533-1589) Latin translation of Dionysius of Halicarnassus' essay on Thucydides, and of a Latin letter written in 1418 by Juraj Jurjević (Georgius de Georgiis, Zadar c. 1400) to Giovanni Battista Bevilacqua.

Edition of Dudić's text refers to local images taken from the archive.org digital facsimile of a 1586 Frankfurt edition. Edition of Jurjević refers both to images of the manuscript (Munich, BSB, Clm 5350) and to (one) passage in Isaiah. We used a Vulgate edition prepared by the Perseus Project, because Perseus uses stable Citation URIs (as developed by Canonical Text Services) for referring to segments of their texts.

Transforming the TEI XML to HTML required slight modification to their set of XSL stylesheets. Technical information about this (written mostly for myself, as I forget it again and again) is here (on klafil dokuwiki).

Filtering Latin words

For anyone speaking Croatian or a host of related languages, "filter" means first and foremost "cigarette filter". There is even a legendary song from the 1980's built around it.

However, in profiling Croatian Latin filters are, more prosaically, ways to save time and resources. Once we have a sufficient set of lemmatized Latin words, we can avoid sending these words to Morphology Service again.

Not one, but three filters are needed. From a list of forms contained in a Latin text (any that we intend to include in CroALa) first will be filtered out all previously unambiguously lemmatized forms. From the remaining set, we'll filter out what was previously recognized, but ambiguously. Finally, a filter will be applied to words previously encountered, but not recognized by the Morphology Service.

What is left is ready for sending to Morphology Service. The resulting JSON will again be sifted into three groups: the lemmatized words, the ambiguously lemmatized, the unrecognized.

E. g. A letter by Juraj Jurjević, a little known nobleman from Zadar interned in Venice in 1418 (Zadar was definitively subjugated by Venice in 1409), consists of 755 words in 536 different forms. The filters separate these forms into 173 previously recognized, 95 previously ambiguously recognized, 268 remaining (now I see that we could have applied the filter for previously unrecognized words, but we didn't do it today).

So 268 Latin forms travelled across the globe to be processed by the Morphology Service on the first day of 2013. Of these forms, 180 were unambiguously lemmatized; there were 139 ambiguous identifications; and 29 forms were listed as forma non recognita. The total score exceeds 268, of course, because of ambiguously identified forms — each of their lemmata gets a row of its own.

Tomorrow I'll write up how all this was accomplished programmatically, in a mix of Bash, Perl, and MySQL.

Morphological JSON with Perl

Learning Perl, aka "the Llama book", makes a terrific didactical point in footnote 8 on page 6:

If you're going to use a programming language for only a few minutes each week or month, you'd prefer one that is easier to learn, since you'll have forgotten nearly all of it from one use to the next. Perl is for people who are programmers for at least twenty minutes a day.

Basically, nulla dies sine linea. The daily twenty minutes today took about three or four hours, but I ended up with Perl version of what I already did in JavaScript: a script that iterates over any list of JSON results from Latin Morphology Service, decides whether a word sent to it has been recognized or not, and then whether the lemmatization is ambiguous or not.

The rizai — getting through all arrays of hashes and hashes of hashes — have been pikrai indeed (the crucial piece of information was shared by this post at Stack Overflow); dereferencing still appears to me as consecutio temporum must look to a programmer; hashes were my Scylla and arrays my Charybdis, but the ship is still sailing, more or less.

The script is here (thanks to DokuWiki).

All this wasn't done as pure exercise (I'm not such a conscientious student). The Morphology Service JSON holds lot more then a lemma, in fact it provides a wealth of information — most of what people interested in natural language processing of Greek and Latin usually lack (and scholars of other languages have). You need to stem a word? You need to identify which part of speech it is? It's all there somewhere, nested deep in JSON.

Naturally, you ask why should I bother. Are we not trained to use dictionaries, don't we have enough grammatical knowledge? Of course we do; we can read Greek and Latin much better than computers. But there are limits to how much we can read, or analyse. Giving the text the care and the gusto it requires — Greek and Latin we have today were not written to be read quickly ‐ I need from one to ten minutes for a page, and enough time for reflexion and rumination afterwards. Grammatical analysis progresses even slower. The computer, on the other hand, doesn't care for rumination; it gets back from Morphology Service JSON for 2000+ words of a neo-Latin text approximately in the time that I need to write this post.

And then we have a chance to learn from computers' mistakes.

Which words were recognized, which are ambiguous, which are unknown to the service? What is the proportion between the three groups? Which words are unambiguously identified, and not inflected? We'll store the uninflected words somewhere, because we don't need to stem them (much); we'll store the unambiguously recognized words, because we won't need to lemmatize them in other texts; from the set of unrecognized words we'll be building an index nominum et locorum, an index verborum rariorum, and a list of common words which Morphology Service should add to its database. Furthermore, a list of lemmata allows us to begin exploring lexical variety in a text, or in a set of texts.

Mind you, the basis for much of this is being put together while I write this. All I had to do to make it happen was learn some code. It almost didn't hurt. Much.

Structuring the Mercurius Croaticus

Mercurius Croaticus is currently a set of TEI XML files containing bibliographical records. These files will be served and made accessible via the BaseX XML database, once we decide on how to present the records; our working premise is that, if researchers want to discover something really interesting, they'll be willing and ready to learn XQuery (not least because it's a powerful tool for more than one research project). And Mercurius Croaticus will help them learn.

To get a clear idea on what exactly is in the bibliographic collection, and to avoid confusion, we organised the files in three sets of folders, following the FRBR concept. There is a folder for authors (auctores), a folder for works (opera), and one for "manifestations" (it is interesting that a term for the third category is not readily available in any language I know); "manifestations" have two subfolders: manuscripts (MS) and printed books (typis edita). Obviously, the "internet" subfolder could also be included.

The folder auctores contains our starting prosopography — a set of 244 personal records for Croatian Latin authors included in the Leksikon hrvatskih pisaca (A Lexicon of Croatian Writers, Zagreb 2000) — and the "additions" file, containing currently 70 more neo-Latin authors of Croatian origin.

Main part of opera is also culled from the Leksikon hrvatskih pisaca — there are 1784 items listed — and there are 18 additional items as well.

The manifestations/MS subfolder has one special collection, with excerpts from Paul Oskar Kristeller's Iter Italicum (no, we don't have the money to subscribe to Brill's internet edition); it was excerpted by Darko Novaković, who kindly lent his notes to Mercurius for TEI XML conversion. And there are records collected from other sources, more or less obiter.

Finally, there is the manifestations/typis edita subfolder, where the basis is the bibliography made by Šime Jurić (1915–2004) in late 1960's: Iugoslaviae scriptores Latini recentioris aetatis (that is, its "Pars I. Opera scriptorum Latinorum natione Croatarum usque ad annum MDCCCXLVIII typis edita. Bibliographiae fundamenta. t. 1. Index alphabeticus. t. 2. Index systematicus. Additamentum I."), later improved by the Croatian National and University Library, and encoded by the Croatiae auctores Latini project. This collection contains 5867 bibliographic records on printed Latin publications with works by Croatian authors. Jurić's bibliography breaks off with the year 1850. Mercurius Croaticus should urgently supplement it with data on later publications, all the way up to the present.

Saturnalia with Perseus Latin JSON

A two-days Christmas project: learn how to use Latin lemmata in JSON format, as provided by the Morphological Analysis Service available on an instance of the Bamboo Services Platform hosted by University of California, Berkeley at http://services-qa.projectbamboo.org/bsp/morphologyservice (used by Perseus for Latin and announced by Bridget Almas on November 1, 2012).

The fanfare: see the results here: [X].

Caveat. If you're a programmer, the following may seem extremely silly, because it is a description of how a non-programmer solved a programmer's task.

The task

What we wanted to do. There is a file of saved JSON responses, produced by sending a list of Latin words (from a Croatian neo-Latin text) to the Morphological Analysis Service, and appending responses. We wanted to produce a simple table containing a form sent to the service and the lemma gotten in response. This has already been achieved locally and indirectly, processing the responses file with some Perl and Linux CLI regex tools to produce a HTML page. But now I wanted to learn how to compute JSON as JSON. Solving the problem also taught me the structure of Morphological Analysis Service response.

The responses file contains lines of JSON objects:

{"RDF":{"Annotation":{"about":"urn:TuftsMorphologyService:abduxerunt:morpheus","hasTarget":{"Description":{"about":"urn:word:abduxerunt"}},"hasBody":{"resource":"urn:uuid:58f0bfcf-0180-4596-92d7-e88eaccffa8b"},"title":null,"creator":{"Agent":{"about":"org.perseus:tools:morpheus.v1"}},"created":"26\nDec\n2012\n12:01:28\nGMT","Body":{"about":"urn:uuid:58f0bfcf-0180-4596-92d7-e88eaccffa8b","type":{"resource":"cnt:ContentAsXML"},"rest":{"entry":{"uri":null,"dict":{"hdwd":{"lang":"lat","$":"abduco"},"pofs":{"order":1,"$":"verb"}},"infl":{"term":{"lang":"lat","stem":"abdu_x","suff":"e_runt"},"pofs":{"order":1,"$":"verb"},"mood":"indicative","num":"plural","pers":"3rd","tense":"perfect","voice":"active","stemtype":"perfstem"}}}}}}} {"RDF":{"Annotation":{"about":"urn:TuftsMorphologyService:abscente:morpheus","hasTarget":{"Description":{"about":"urn:word:abscente"}},"title":null,"creator":{"Agent":{"about":"org.perseus:tools:morpheus.v1"}},"created":"26\nDec\n2012\n12:01:28\nGMT"}}} {"RDF":{"Annotation":{"about":"urn:TuftsMorphologyService:abstineant:morpheus","hasTarget":{"Description":{"about":"urn:word:abstineant"}},"hasBody":{"resource":"urn:uuid:566cf4ec-2a8c-452f-a02f-5e0cecf32f52"},"title":null,"creator":{"Agent":{"about":"org.perseus:tools:morpheus.v1"}},"created":"26\nDec\n2012\n12:01:29\nGMT","Body":{"about":"urn:uuid:566cf4ec-2a8c-452f-a02f-5e0cecf32f52","type":{"resource":"cnt:ContentAsXML"},"rest":{"entry":{"uri":null,"dict":{"hdwd":{"lang":"lat","$":"abstineo"},"pofs":{"order":1,"$":"verb"}},"infl":{"term":{"lang":"lat","stem":"abs:tin","suff":"eant"},"pofs":{"order":1,"$":"verb"},"mood":"subjunctive","num":"plural","pers":"3rd","tense":"present","voice":"active","stemtype":"conj2","morph":"comp_only"}}}}}}}

The JSON

"Lines of JSON objects" is not a valid JSON, as you can see if you copy the lines above and paste them here: jsonlint.com. Why the error? All objects have to be contained in a JSON array. Also, for some reason "RDF" wasn't accepted as field name (key). So we transformed the file locally, introducing the "Verba" as the top array key, like this:

perl -0777 -wpl -e "s/\n/,\n/g;" jsonfilename | perl -wpl -e 's/"RDF"/"rdf"/g;' | perl -0777 -wpl -e 's/^{/{"Verba":\[{/;' | perl -0777 -wpl -e 's/,\n*$/\n\]}\n/;' > resultfilename.json

To process the JSON, we used JavaScript (amazed by the d3 JavaScript library). D3 documentation and examples showed how to read in a JSON file — and then I realized that the Morphology Service JSON is pretty complicated:

• it is (sometimes deeply) nested
• its objects contain arrays
• an unsuccessfully lemmatized word won't have any "Body" object; a lemmatized word will have a string there; for an ambiguously lemmatized word, "Body" will be an array

The loops

So, lots of loops here. First the script had to loop through the array:

var infoLength= data.Verba.length; for (infoIndex = 0; infoIndex < infoLength; infoIndex++) { // ... }

Then it had to check whether a JSON object contains a lemmatized word — whether it had "Body" object. The a1 variable will hold the form sent to the service, while the a2 will hold the lemmata (or information that the word wasn't lemmatized successfully):

var a1 = data.Verba[infoIndex].rdf.Annotation.hasTarget.Description.about; // testing for existence of lemma var verbdata = data.Verba[infoIndex].rdf.Annotation.Body; if (verbdata) { // ... } else { var a2 = "FORMA NON RECOGNITA"; }

And then the "Body" object had to be tested for array; in case it isn't an array, JSON would be traversed all the way to the "$" key (containing the dictionary entry for the lemma):

if(Object.prototype.toString.call(verbdata) === '[object Array]') { // Iterate the array and do stuff ... } else { var link2 = perseus + data.Verba[infoIndex].rdf.Annotation.Body.rest.entry.dict.hdwd.$ ; var a2 = data.Verba[infoIndex].rdf.Annotation.Body.rest.entry.dict.hdwd.$ ; }

Now, in case that the verbdata variable contains an array, the array had to be iterated over, and a list — actually, a new array — had to be built from its values:

var a2 = []; for (bodyIndex = 0; bodyIndex < verbdata.length; bodyIndex++) { a2.push(verbdata[bodyIndex].rest.entry.dict.hdwd.$); }

Finally, we used a small routine to populate a table with resulting forms / lemmata pairs:

var columns = [a1, a2]; var table = d3.select("#container") .append("tr") .selectAll("td") .data(columns) .enter() .append("td") .text(function(column) { return column; });

Lots of trial-and-error (Firebug was a great help, and Stack Overflow even greater one) need not be dwelt on. Just one limitation puzzles me: the JSON file contains responses on more than 2000 words; my version of Firefox throws an error after ca. 720 objects read — either something should be optimized, or a paging system introduced. And, of course, seeing all 2000+ forms/lemmata pairs at once is neither necessary nor useful; the only thing we need is an option to sort out the unrecognized forms. This was added by the sorttable.js script.

Once again, the page with our JavaScript can be seen in action here: [X].

One-line concordance in Linux command line

A recipe. To create a "concordance" — actually, a list of forms from a text with frequencies added — using just a command line, skipping programs such as AntConc (which is great, nice and illuminating, but sometimes I just need to prepare a list quickly). It can be done with the following Bash one-liner:

tr '[:punct:]' ' ' < filename1 | tr '[:upper:]' '[:lower:]' | tr '[:blank:]' ' ' | sort | uniq -c | sed 's/ \{1,\}/","/g' | sed 's/^",//g' | sed 's/$/"/g' > filename2.csv (Filename1 is input file, filename2.csv output in csv format.)

Recently there was a discussion on HUMANIST list whether "bash scripting is a worthwhile approach to "tool" development in the Digital Humanities". People tended to reply no, either learn a "real" language (Python was recommended), or develop a GUI ("like most people, humanists have a strong distaste for the commandline"); I think it was 3:1 against the command line.

Obviously, I disagree. Using bash helped me cross the boundary between user and "programmer" — on the command line one just slides from one region into another. Without a formal education: you have a problem, you look for a solution (discovering gratefully that you stand on shoulders of many colleagues), and bam! it's solved.

I think that digital humanists in general should adopt this kind of sliding — from users to programmers as well as from "classical" to "avant-guarde" scholars — as their MO.

Asterisks

* * *

Today I had to mark up several Latin poems which used the device above -- three asterisks, even placed as an asterism -- to mark breaks between thematic units.

How to mark a set of asterisks, a typographical asterism, in TEI XML? A TEI-L discussion from 2007 helped, and I decided to use the space element ("indicates the location of a significant space in the copy text").

⁂

Profiling cultural literacy of Croatian Latin writers

A paper to be presented in the Latin, National Identity and the Language Question in Central Europe conference, organised by the Ludwig Boltzmann Institute for Neo-Latin Studies in Innsbruck (12--15 Dec 2012) will apply E. D. Hirsch's concept of cultural literacy --- which is actually German Allgemeinbildung or Bildungsgut (see it at work in these books), and "opća kultura" in Croatian --- to intellectual horizons of Croatian neo-Latin writers, as represented in the Croatiae auctores Latini collection.

Judging from the programme, the conference offers a chance to present digital research to an audience working mostly in "traditional" ways. This is quite an opportunity; too often digital humanities get separated in a room of their own, where they don't get in anybody's way. So, the challenge is to persuade colleagues that a large-scale search of CroALa using e. g. common terms from CAMENA TERMINI can lead to something interesting.

Update, post-conference, 19/12/2012:

My presentation on cultural literacy is here (note to self: never again try to do a presentation in a browser — outdated browser versions always turn up in crucial moments in crucial spots).

The paper itself is here.

Preparing to launch the Mercurius Croaticus

Mercurius Croaticus, a prosopographical and bibliographical collection of data on Croatian Latin writers, their works, editions, and manuscripts, is nearing its launching point.

The starting dataset comprises information on:

• 269 authors


• 1808 works


• 5867 printed editions


• 58 manuscripts (more to be added soon)


• digitised copies, where available

Beneša's trigrams

Here is an experimental page for researching trigrams in the De morte Christi, a neo-Latin epic by Damianus Benessa (Damjan Beneša).

What did we do:
1. using a concordance program (our reliable AntConc), we found trigrams in Beneša's Latin text, which we obtained courtesy of our colleague Vlado Rezar, Beneša's modern editor

2. we reformatted the trigrams slightly, using tr and sed, to make use of the excellent PhiloLogic crapser function (it is hard not to laugh thinking about this function, because in Croatian "serem" means "I crap")

3. using curl and a simple bash script, we sent the trigrams to CroALa

4. using again sed, we filtered out the successful hits, i. e. those which produced results

5. with some more sed, the hits were turned into searches on the page linked to at the beginning: [X]. There you'll find the trigram which produced the hit, the link to a saved search, and a report on the number of occurrences found in CroALa.

Most interesting findings for us are occurrences from Marko Marulić and Jakov Bunić, close contemporaries of Beneša; Marulić and Bunić also wrote Biblical epic poems in Latin (and Marulić's epic remained in manuscript until the 1950's).

The useful sed snippet which produces the regex line, and a line immediately before it, is here:

sed -n '/Your search found/{x;1!p;g;$N;p;};h' ben-filename

(Adapted from that goldmine, the Sed one-liners.)

Thinking about PND

An important part of our research is finding the Personennamendatei (PND) number of Croatian Latin authors and adding the number to our personal data record of the author. So far, 83 authors (of 241 from our experimental set) have been connected with their PND-Nrs.

Now we're looking into the ways Wikipedia (at least, German Wikipedia) explores the PND to uniquely identify persons and connect data on them. The BEACON format seems a nice start for a small catalogue like ours. And, of course, it would be nice if Croatian Wikipedia decided to adopt something similar to the PND scheme.