Thus Spake the Drum
“Come Back Home”
Make your feet come back the way they went,
make your legs come back the way they went,
plant your feet and your legs below,
in the village which belongs to us.
In the first half of the 18th century, European colonizers noticed that each tribe of Sub-Saharan Africa had worked out long distance communication a long time ago and used it effectively. On the other hand, in Europe, they were still figuring out reliable long-distance communication that works faster than any traveler on foot or horseback. They had tried flags, horns, intermittent smoke signals, flashing mirrors even!
How did they do it? The “primitives” of sub-Saharan Africa had figured it all out! They communicated not just simple transactional messages, but jokes, stories, and poetry. How did they make speech and communication work?
Language is an interesting and a formidable beast. Tribes in Africa had tamed it to suit their purposes. Of course, Europeans did come up with Morse Code soon. But Morse Code is not language, it’s code. (Well, duh!) Codes are efficient, whereas languages are not. Languages are full of redundancies.
Claude Shannon was the first to think mathematically about this, which led to its formalization as Information Theory. This is how he opens his seminal paper of 1948 “A Mathematical Theory of Communication”
“The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning.”
For some reason I’ve always found the above profound and amusing in equal measure — especially the last sentence.
There have been numerous advances in information theory, machine learning, and natural language processing (NLP) since then.
In the third talk of the Representations series, Language Models or “Oh fork you, autocorrect!” I’ll discuss Language models (LMs). LMs capture the statistical nature that’s inherent to languages. What’s really cool about LMs is that they can be trained in an unsupervised manner. And yet they are the staple for almost any NLP task starting from something as basic as spell checkers to the holy grail of machine translation.
Language Models aren’t a new tool. But their importance has grown with time. In fact, some of the research work in the last few years has been so pivotal, there are claims that the “ImageNet moment for NLP has arrived” (I’ll explain what this means and its importance during the talk.)
I’ll have more to say about LMs later. But let’s start with the basics. Let’s start by discussing a basic type of LMs called n-gram language models, and work through building a simple spell corrector.
If this sounds interesting to you, feel free to attend my talk tomorrow. Check the event details and indicate your interest here: https://www.linkedin.com/events/talkseries-representations-lang6760975829481746432/