This directory contains a suite of software used to index and then search databases of sentences, and its purpose is to facilitate a question/answer interface to Distant Reader study carrels. For example, you could ask the system "Who killed Hector?" and get back an answer something like this:

Alternatively, you could ask the system "What is knowledge?", and get back something like this:

Very important! This system is not intended nor expected to return the answers to given questions. Instead it is intended to return plausible answers, and you are expected to use the results as discussion points or to use traditional reading techniques for the purpose of verification.

Think of this system as a tool to supplement your reading. Create a collection of texts, index (model) the collection, search the index, interact with the model, and in the end, garner a better understanding of the collection. Think of the whole process as a sort of interactive discussion with a book. As such, this system implements a form of reading.

First, you will need to install the Distant Reader Toolbox, and if you get past this step, then the rest ought to be relatively easy (famous last words):

Then you will need to create and/or download at least one Distant Reader study carrel. For example, download the Iliad and the Odyssey by Homer:

Next, you will need to install the Reader Sentences scripts:

(Because zero websites, let alone URLs, are permanent, I have cached the Reader Sentences software at https://distantreader.org/blog/reader-sentences/reader-sentences.zip, just in case.)

Change directories to the just downloaded repository:

Now, read and cache all the sentences in a given study carrel:

At this stage, it is quite likely there will be missing Python modules. Do you best to install them.

The next step is to vectorize ("index") the sentences:

Again, it is quite likely you will have missing Python modules and/or you do not have Ollama installed. Do your best to install Ollama. You will also need to install two large language models: 1) nomic-embed-text, and 2) llama2. First, nomic-embed-text:

And then llama2:

Repeat the previous steps, and please be patient; these steps are computationally expensive.

Once you get this far, you can query the database of vectorized sentences. The following command queries the study carrel named "author-homer-gutenberg" for the word "hector" and returns thirty-two sentences:

The result ought be a long paragraph thirty-two sentences in length. Each sentence ought to allude to Hector in some way, shape, or form.

One way to make more sense of the long paragraph is to divide it into smaller paragraphs, like this:

Another way to make more sense of the long paragraph is to use a large-language model to summarize it:

Finally, you can use the following command to actually submit a question to be addressed by the system. For example:

This suite of software is made up many little Python scripts and Bash front-ends. This first list of scripts are the most used:

The following scripts are front-ends to their Python equivalents, and all they really do is add some formatting to the outputs:

Queries can be of just about any length and require zero syntax. That said, it is oft-times difficult to articulate useful, meaningful, or comprehensive queries. The scripts below use extracted features from the given study carrel to create more expressive queries for you:

In natural language processing, a set of stop words is a list of words with no or little importance. Examples usually include the words "the", "a", "an", "of", etc. Conversely, one might articulate a list of very useful words -- words of great significance. Such a set of words is sometimes called a "lexicon". If you create a file named ./etc/lexicon.txt within your study carrel(s), then the following scripts will use that file as they query part of the input:

The curation of lexicons is a thing all unto itself. See Reader Lexicons for ways to create more expressive lexicons.

The following two scripts help you to define words. They do not output thedefinitions of words but rather plausibledefinitions:

The following are miscellaneous scripts:

The following scripts are just for fun. They employ a Markov modeling technique to pseudo-randomly generate sentences. Use these scripts to become familiar with the common bigrams (two-word phrases) in the given carrel.

First, you must ask yourself some sort of question. Given a study carrel, what do you want to know? The questions you ask can range from the mundane to the sublime, but you must ask yourself a question. In this case, my question will be, "In Homer's Iliad, who is Hector?" And to begin with, I will download a study carrel, cache it's sentences, and vectorize them:

Again, please be patient. The vectorizing process is computationally expensive. On my Macintosh laptop with eight cores, the vectorizing process takes about five minutes. Your milage will vary.

The next step is to begin querying the study carrel, and I always suggest starting out very small. Consequently I will query the study carrel for a single word and request a single sentence:

I get the following result, it is merely the sentence which is most like the query. Apparently Hector, who ever he it, dies:

I then increase the number of sentences to return, and I suggest you always double the value. Thus, the following command identifies the two sentences most closely matching the query:

My result:

Apparently, Hector, who ever he is, has a brother.

Get some more sentences, and again, I suggest doubling the desired number of sentences:

Apparently, Hector is brave.

Yet more:

Apparently, Hector is angry and he uses spears.

Even more:

Hector seems to be the son of Priam, has fury, does mischief, and rides chariots.

Search for many sentences:

By this time the results are long and somewhat difficult to consume, but you can reformat the results by transforming the big paragraph into smaller ones. The result is easier to read:

Not incidentally, the format command computes the cosine distance between each sentence and its subsequent sentence. If the distance is great, then we can assume the subsquent sentence is not like the given sentence, and therefore a new paragraph is begun.

Moreover, sentences are not returned in any sort of relevancy ranking nor closeness value. Instead, sentences are sorted by the name of the study carrel from whence they came and then by the ordinal number of the sentence. Consequently, sentences which are closer together in the output are more likely to appear in the same study carrel item.

One could now closely read the formatted output, and one can begin to answer the question, "Who is Hector?" On the other hand, you can supplement the process by summarizing it with the use of a large-language model:

Given enough context, as per above, one can now elaborate on an idea and ask a specific question:

At the very beginning of this reading, it was brought to our attention that Hector dies. Well, let's ask how:

Let's now try to figure out how Ulysses and Penelope are related. Like before start out small:

Apparently they know each other.

Get more context:

She recognizes him.

Get even more context:

Apparently Penelope is admirable and has an excellent nature, and if you read closely Ulysses calls Penelope "wife":

Jump ahead and get a lot of context, reformat the results, and summarize:

Finally, ask a specific question:

A few possible next steps include:

This suite of software is as tool for reading. Create and/or download Distant Reader study carrels ("data sets"). Extract and cache all of the sentences in all of the items in the carrel. Vectorize (index/model) them. Use the scripts in this distribution to query the sentences and thus become familiar with the carrel's content. Compare and contrast the results from one study carrel with the results from another study carrel. Use the outputs of these scripts as points of discussion with other people.

Finally, while the system uses various techniques to riff on the search results, do your best to closely read the search results before you summarize or elaborate. It is your responsibility to figure out the degree any of the underlying large-language models are hallucinating. Again, the system does not output the answer but instead it outputs plausible answers.