QuestionQ56

VerifiedInitial Analysis of the Data

Consider this text:

“Stop!” he shouted. “Don’t go there!”

Which set of words results when a punctuation tokenizer is applied to the text?

Explanation

A punctuation tokenizer uses punctuation as token boundaries. The apostrophe in “Don’t” separates it into “don” and “t”, while the other punctuation is excluded from the resulting words.

Community Discussion

No comments yet. Be the first to start the discussion!