nltk on C'est la Z

Working with texts part 3 - word chains

Mon, 25 Nov 2019 05:45:55 -0400

At this point, we've done a fair amount of playing with text so it's time for a fun little project. We're going to generate some text "in the style" of a source text. The technique we're going to use is usually called a Markov Chain text generator. Basically a model where the next state or word is based entirely on the current state. I don't dwell on the math under the hood but in case you're interested, here are a few links: Wikipedia, Explained Visually, UC Davis Math.

You can motivate this lesson in a number of ways. One is to start by showing some generated text - for this, a flowery, poetic text like the Book of Psalms works well. You can generate text a bunch of times until you get a good result.

Here's the start of the Book of Psalms:

Blessed is the man that walketh not in the counsel of the
ungodly, nor standeth in the way of sinners, nor sitteth in
the seat of the scornful.

and here's some generated text:

Blessed is the people that delight in war. Princes shall come out of
Zion, the perfection of beauty god hath shined our god shall come and
shall declare thy mighty acts. I will speak of the glory of the lord
surely i will not lie unto David.

I added the punctuation as our generator strips it all and converts the text to lower case.

The generated text looks reasonable in some ways but is clearly wrong in others.

This leads to the conversation.

If we look at the original text, it starts with "blessed" but if we look at our bag of words we also can see that "blessed" appears 49 times. What's interesting for generating text are the words the follow "blessed" and the number of times they follow it. Here they are (sorted for convenience):

Word How many times it follows the word "blessed"

are 5

art 1

be 16

blessed 1

depart 1

for 1

his 1

in 1

is 14

of 2

that 1

thee 1

thy 1

upon 1

wealth 1

you 1

Word	How many times it follows the word "blessed"
are	5
art	1
be	16
blessed	1
depart	1
for	1
his	1
in	1
is	14
of	2
that	1
thee	1
thy	1
upon	1
wealth	1
you	1

So we'll have a lot of things like "blessed be the" "blessed be he" etc. but only one occurrence of "blessed upon…." To generate text, we can build a dictionary where the keys are the words and the values are all the words that follow the key. We can build this with a variation on our bag of words builder:

 words_raw = "a cleaned string with our text"
 words = words.raw.split()
 wordlists = {}
 for i in range(len(words)-1):
 current_word = words[i]
 next_word = words[i+1]
 wordlists.setdefault(current_word,[])
 wordlists.[current_word].append(next_word)

Note that we need a counting loop rather than just iterating over the list since we need both the current item and the one after it in each iteration.

The (partial) value for the wordlists for the Book of Psalms would be:

 wordlists = {...
 'blessed' : ['are', 'are', 'are', 'are', 'are', 'art', 'be', ...],
 'wealth' : ['and','and','by','to'],
 ...
 }

At this point, generating text is easy.

Start with a word
Add the word to yoru story.
Look it up in your wordlists dictionary.
select a random word from that words dictionary entry (it's list).
That becomes the word for the next iteration
Repeat until you have enough words.

This won't add any punctuation and it will crash when it gets to the last word in the original text (as it can't find any "next words" but basically it will work. Here's the code:

 # assume wordlists is the dictionary built as specified above
 # and that we're starting with the word "blessed"
 current_word = "blessed"
 storylist = []
 for i in range(storylength):
 story.append(current_word)
 current_word = random.choice(wordlists[current_word])
 story = " ".join(storylist)

The random.choice() randomly selects a word from a list. This is perfect for us since words that follow our key more frequently will have more entries in the list and thus appear more often. I decided to build a list of words and then use the " ".join() to turn it all into a space separated string.

This is pretty fun but the students will note that it really doesn't work that well. This can lead to increasing our sample. Instead of using a single word as our key and having chain links based on two words (key, random choice from value), we can use three word links. Use a two word tuple for the key and the same list of words as the value.

For example, a partial dictionary based on this idea from Macbeth Act 1 Scene 1 could be:

 # source -> "when shall we three meet again"
 tuple_dict = { ("when","shall") : ["we"],
 ("shall","we") : ["three"],
 ("we","three") : ["meet"],
 ("tree","meet") : ["again"]}

This takes a bit more work to build and also a bit more work to generate text but it's eminently doable and the result is better. What about three word keys? Four? These aren't hard to write and ambitious students can write a generic dicitonary builder and story generator routine so as to be able to use any key lenght.

Students will find that the longer the key, the better the resultant story. They'll also notice that at some point, all you'll ever generate is the original back again. This is a great time to talk about over-training.

This can be a fun unit and / or assignment. I used source materials that I like but any text will work.

Working with texts part 2 - bag of words

Wed, 20 Nov 2019 08:15:41 -0400

Following up on a previous post, we're going to continue to talk about playing with text. This time, building and working with a bag of words from a text.

A bag of words is a simple language processing model where you just consider individual words in a text. What they are and how many times they occur. This is a pretty simple model but you can still have a good bit of fun with your students with it.

For example, if you take a text like this:

This land is your land this land is my land

Once you clean the a data as per the previous post you get the following bag:

word	Count
is	2
land	4
my	1
this	2
your	1

Not too interesting but it gets more interesting on a larger text. We used:

Act 1 Scene 1 from Macbeth (for testing - it's only 13 lines)
Cyrano de Bergerac (because it's my favorite play)
Moby Dick (chapter 1)
Moby Dick (full text)
The Book of Psalms

Assuming a cleaned string with all the text, building the bag is pretty easy:

1cleaned_text = "a string with all the cleaned text"
2bag={}
3for word in cleaned_text.split():
4 bag.setdefault(word,0)
5 bag[word] = bag[word] + 1

The setdefault in line three says that if word isn't a key in the dictionary then insert it with a value of 0, otherwise, if word is already a key in the dictionary do nothing. This saves us from constructs like:

1 if word not in bag:
2 bag[word]=1
3 else:
4 bag[word] = bag[word] + 1

Once we have our bag we can do some explorations. Take out all the values using bag.values() and sort them. Ask "what words are going to occur most frequently?" It's easy to guess that it will be words like the, of, and etc.. Words that don't really tell us anything about the text. It's a fun puzzle to figure out where the "interesting" words start and even the question of what might make a word in a text "interesting." Words like the, *and", etc. are known as stop words. Depending on the application, you might want ot remove them. Then again, you might not.

Usually we take a bit of time just to play with the bag, looking at the types of words in various count ranges. Sometimes we'll make bags for different chapters of books and compare them. This is also when I cover Python list comprehensions which make experimenting on bags of words much cleaner. Say, for instance you want to find all the words that occur between 50 and 100 times. Without list comprehensions you might write:

1result = []
2for word in bag:
3 if bag[word] >= 50 and bag[word] <= 100:
4 result.append(word) # or (word,count) or whatever

While with the comprehension it's:

result = [ word for word in bag if bag[word] >= 50 and bag[word] <= 100]

Besides basic exploring, what else can you do? You can make word vectors, that is a list or vector where an element is 1 if the word is present and 0 otherwise. Then you can do simple document comparisons. You can kick this up a notch if you calculate word frequencies based on the bag counts and use these.

You can also do basic sentiment analysis. Download a list of "positive" and "negative" words and then check bags against them. You can check to see how many negative or positive words are in a bag or you can work with frequencies. You can also use other categories.

While we were discussing this at our PD, some teachers thought that their classes might ahve a hard time doing all of this from scratch but thought that they could provide either the cleaning code or the bag building code and have the students build experiments off of the base. I thought it was a great idea.

At the end of our session we all agreed to think about how we might build a lesson or unit out of this bag of words idea. I'll leave you all with some web sites that use these ideas in what I think are interesting ways:

Enjoy!