By Sagi Shaier · 22 October 2026 · 9 min read

What is NLP? Natural language processing explained

The main NLP tasks from sentiment analysis to language modeling, the kinds of ambiguity that make human language hard for a computer, and why modern NLP learns from data instead of following hand-written rules.

NLP (natural language processing) is the field of getting computers to process, understand and generate human language. Web search and autocomplete are NLP systems. So are translation apps, voice assistants, spam filters and chatbots, each one a computer doing something useful with text or speech.

What "natural" language means

In NLP, "natural" means human languages such as English or Mandarin, which developed over time through everyday use. The contrast is a formal language like Python, SQL or math notation. A formal language is designed with exact syntax, so a parser (the program that reads it) can recover one valid structure for each line.

Human language works differently. It has exceptions, words whose meaning depends on context, and sentences that can be read in more than one way. "I saw her duck" can mean she owns a duck or that she ducked down, and nothing in the sentence settles which.

Natural language, developed through everyday use, shown with "I saw her duck" whose meaning depends on context, next to formal language, designed with precise syntax, shown with x = duck() where the parser expects one structure.

Why NLP is useful

A large share of human knowledge is stored as text such as books, emails and medical notes. There is also far more of it than any team of people could read. A system that reads and organizes text at that scale can do work that would otherwise not get done, like sorting millions of clinical notes or support tickets.

Language is also the easiest way for most people to tell a machine what they want. Typing or saying a request directly is simpler than learning a menu structure or a query syntax, which is why chat interfaces and voice assistants reach so many people. In each case, language goes in, and language or a decision comes out.

Five NLP inputs and outputs: a review becomes a positive sentiment label, a page becomes a Japanese translation, an invoice becomes structured fields, a question becomes the answer Tokyo, and a prompt becomes a written paragraph.

Large language models made NLP much more visible. Systems trained to predict text turned out to be useful for conversation, writing and coding. For many people they are now the first place they meet machine learning.

Understanding vs generation

NLP splits into two broad halves. Natural language understanding (NLU) is the reading side: the input is text and the system has to interpret it, for example by finding the sentiment of a review or who a pronoun refers to. Natural language generation (NLG) is the writing side: the system produces text, like a translation, a summary or the next word in a sentence.

NLU drawn as text going into a system that reads topic, sentiment and what "it" refers to, and NLG drawn as a system producing text such as a translation, a summary or the next word.

Many systems do both. A translator has to understand the source sentence before it can generate the target. The line blurs further in modern systems: a large language model asked whether a review is positive does an understanding task, but it answers by generating the word "positive".

What an NLP task is

NLP organizes itself around tasks. A task is a specific mapping from an input to an output, with both sides stated up front. For example, sentiment analysis takes a piece of text and outputs one label: positive, negative or neutral. Once a task is stated that cleanly, you can collect examples, train a model and measure how often it is right. Two systems can then be compared on the same task with the same scoring.

Three tasks drawn as mappings: a movie review to the label positive, "The cat sat on the mat." to its French translation, and "What is the capital of Japan?" to the answer Tokyo.

The common task families:

Task	Input	Output
Text classification	a whole text	one label, like a sentiment or topic
Sequence labeling	a sentence	one label per word
Machine translation	text in one language	text in another
Question answering	a question, often with a document	an answer
Summarization	a long text	a short version
Information extraction	unstructured text	structured records
Natural language inference	two sentences	entail, contradict or neutral
Language modeling	the words so far	the next word

Text classification gives one label to a whole text. Examples are sentiment analysis, topic classification and intent detection, where "cancel my order" maps to the cancel action. Sequence labeling gives a label to each word instead. Named entity recognition (NER) tags words that name people, places, organizations and dates. In "Maria flew to Lisbon on Friday," it marks Maria as a person, Lisbon as a location and Friday as a date. Words that are not part of any name get the label O.

Sequence labeling on "Maria flew to Lisbon on Friday": Maria tagged PERSON, Lisbon tagged LOCATION, Friday tagged DATE, and flew, to and on tagged O, one tag per token.

Question answering comes in two styles. Extractive QA copies the answer as a span straight out of a source document, and generative QA writes the answer in its own words. Summarization has the same split: extractive summaries stitch together the most important sentences, and abstractive summaries rewrite the content in new words. Information extraction turns prose into rows and fields, for example pulling the buyer, the company bought and the price out of a news story about an acquisition.

Some tasks compare two sentences. In natural language inference (NLI), the system decides how a premise relates to a hypothesis. The premise can entail the hypothesis (if the first is true the second must be), contradict it or leave it neutral. From "the band played for two hours," the hypothesis "a band performed" is entailed. "The band never played" is a contradiction and "the band was famous" is neutral. Paraphrase identification decides whether two differently worded sentences mean the same thing, like "How do I reset my password?" and "What's the way to change my login?"

Language modeling, the task under many others

One task sits under many modern NLP systems: language modeling. A language model predicts language from language. The classic version predicts the next word (or the next token, a word or a piece of a word) given the words before it. If the input is "the cat sat on the," a good model gives high probability to "mat" and low probability to "airplane."

Language modeling on "the cat sat on the ?": the model gives mat 0.41, floor 0.22, roof 0.15 and couch 0.09, and text is generated one word at a time by feeding each prediction back in.

Generating text works by picking a next word, appending it to the input and predicting again. Many other tasks can be framed the same way. Translation becomes generating the target sentence, summarization becomes generating a shorter version, and question answering becomes generating the answer. A model trained to continue text can handle many tasks when the prompt describes the task.

Why NLP is hard

The central problem is ambiguity: language leaves a lot unsaid, and the same words can carry several meanings. People resolve it with context, shared knowledge and expectations. A model has none of that built in and has to infer the likely reading from the sentence, the text around it, and patterns it learned from data.

The sentence "I saw her duck." with two readings: duck the bird and duck as in crouch, with nothing in the sentence deciding between them.

Ambiguity shows up in several forms:

Lexical ambiguity is one word with more than one meaning. "Bat" is a flying animal and also the thing you swing at a baseball. Choosing the intended meaning is its own NLP task, called word sense disambiguation.
Syntactic ambiguity is when the words can be grouped in more than one valid way. In "The girl hit the boy with the book," she either used the book or hit a boy who was holding one, and both readings follow the grammar.
Referential ambiguity is deciding what a pronoun points to. In "I poured the water from the jug into the glass until it was full," "it" is the glass. Change "full" to "empty" and "it" becomes the jug.

The jug and glass sentence twice: with "full", "it" points to the glass, the container being filled, and with "empty", "it" points to the jug, the container poured from, so everyday knowledge decides the answer.

The grammar is the same in both versions of the jug sentence. What changes the answer is everyday knowledge about pouring, which the sentence never states. Meaning also depends on who is speaking and why. "Nice job" can be praise or sarcasm, and "Can you pass the salt?" is a request, not a question about ability. This gap between what is said and what is meant is called pragmatics, and a system that reads only the surface words can take a sarcastic review as a glowing one.

Two more properties make it harder. People constantly produce sentences that have never been written before, so a system cannot work from a stored list of valid sentences and has to handle inputs it has never seen. And word frequencies follow Zipf's law: a handful of words like "the," "of" and "and" are extremely common, while a long tail of words appear only a few times even in a huge collection of text. However much text a model trains on, it keeps meeting words it has barely seen.

Why modern NLP uses probabilities

All of this makes hard yes-or-no rules a poor fit for open-ended language. A rule either matches or it does not, while language often calls for a graded judgment: "bat" is probably the animal in this sentence, or this grouping of words is more likely than that one. Modern NLP models assign a probability to each reading and pick the most likely one.

A rule gives a single yes or no, while a probability ranks the readings of "bat": the animal 0.72, the equipment 0.21, to blink 0.05 and other 0.02.

The role of machine learning in NLP

Language has too many patterns and exceptions to write out by hand. English plurals are a small example. Start with "add -s," then handle "-es" and "-ies," and the irregular words still break the rules:

1def plural(word):
2    if word.endswith(("s", "x", "z", "ch", "sh")):
3        return word + "es"
4    if word.endswith("y") and word[-2] not in "aeiou":
5        return word[:-1] + "ies"
6    return word + "s"
7 
8for w in ["cat", "bus", "city", "child", "mouse", "sheep"]:
9    print(w, "->", plural(w))
10# cat -> cats
11# bus -> buses
12# city -> cities
13# child -> childs
14# mouse -> mouses
15# sheep -> sheeps

Each exception needs another rule, and that is one narrow corner of one language. Machine learning turns it around: show the system enough examples and let it find the patterns, exceptions included.

Hand-written plural rules with a growing list of exceptions like child to children and mouse to mice, next to a model that learns from examples and handles a new input like octopus to octopuses.

The most direct version is supervised learning. You collect text annotated with the answers you want, such as sentences paired with their sentiment, and train a model to reproduce those answers. The model outputs probabilities, which is how it copes with ambiguity, and a well-trained model generalizes: it makes sensible predictions on sentences it never saw in training.

Modern NLP pushed this further with self-supervised learning, where the raw text creates its own training signal. A model hides part of a sentence and learns to predict it, either a masked word in the middle or the next token at the end. The answer is already in the text, so no one has to label anything and the training data can be enormous. This is one reason large language models can be trained on so much text and then adapted or prompted for specific tasks.

Two self-supervised tasks on "The cat sat on the mat.": predicting the hidden word mat from [MASK], and predicting the final period as the next token, with the target taken from the raw text both times.

Common misconceptions

NLP means chatbots. Chat is one application. Search, spam filtering, translation and sentiment analysis are NLP too.
NLP and computational linguistics are the same field. They share methods and often the same people. NLP is engineering, aimed at systems that do something useful with language. Computational linguistics uses computation to study how language works, and its goal is understanding.
Enough data removes rare words. Zipf's law means the long tail of rare words remains in any collection, so a model has to handle words it has barely seen.
Understanding and generation are separate systems. Many modern systems do both, and a large language model delivers even a classification label by generating text.

NLP sits where three fields meet. Linguistics describes how language works: its sounds, word structure, grammar and meaning. Computer science brings algorithms and data structures. Machine learning brings methods that learn patterns from data instead of following hand-written rules. For the order to learn the rest of NLP in, see the NLP roadmap.

QuiddityML teaches these ideas in the first unit of its Intro to NLP track. The exercises include matching each input and output shape to its task type, working out what "it" refers to in the jug and glass sentence, and explaining why learning from examples beats hand-written rules for English plurals.