1.1.1 Litterals and metacharacters

In the above examples, the pattern is composed of litterals, i.e., characters that recieve a literal interpretation in the regular expression. Some other characters, on the other hand, receive a different interpretation when they are part of the regex. This is the case of the following reserved characters, called metacharacters: . \ | ( ) [ { $ * + ?. If we want these characters to be literaly interpretted, we need to escape them. In R, this is done using two backslash.

# The character `.` is a metacharacter that matches any character (except a new line)
str_detect(string = c("Earthquake.", "Earthquake"), pattern = ".")

## [1] TRUE TRUE

# To look for a dot in a string:
str_detect(string = c("Earthquake.", "Earthquake"), pattern = "\\.")

## [1]  TRUE FALSE

1.1.2 Line anchors

To match the beginning and the end of a string, respectively, we can use the line anchors ^ and $.

For example, to look for the tweets which begin with a hashtag:

str_detect(string = tweets_earthquake$tweet_text,
           pattern = "^#") %>% 
  which() %>% 
  head()

## [1] 10 25 28 31 38 43

This can be useful when combined with the function dplyr::filter() to filter rows of a two dimension table:

# Tweets beginning with a sharp
tweets_earthquake %>% 
  filter(str_detect(string = tweet_text, pattern = "^#")) %>% 
  select(tweet_text)

## # A tibble: 816 x 1
##    tweet_text                                                              
##    <chr>                                                                   
##  1 #earthquake @BBCNews my uncle is travelling in Nepal but has notified u…
##  2 #Kathmandu's Tribhuvan Airport is currently closed due to #lEarthquake.…
##  3 #Nepal earthquake claims five lives in East #India http://t.co/aU4EFiKU…
##  4 #google person finder for #earthquake http://t.co/uZyXguoio2            
##  5 #This Is the #helpline for #Nepal earthquake click on this post  https:…
##  6 "#\xbe\xdc\xdd\x8c_\xc9 #\x8c\xe0\xbc\x8a__\x8b\x81\xe3\x8d_\xc8 #\x8b\…
##  7 #NepalQuake | Deep Kumar Upadhyay, Nepal's ambassador to India, says Ai…
##  8 #BeingIndian mourns the loss of the lives in the #earthquake that hit N…
##  9 #NepalEarthquake Tribhuvan Int. Arprt #Kathmandu closed 4 operations fl…
## 10 #earthquakeindia Judging by the nature of tremors in Lakhimpur,one can …
## # … with 806 more rows

1.1.3 Alternation

The pipe character | allows to match one or more expression.

tweets <- 
  c("PANIC IN NEPAL: Strong quake hits capital, causing major damage, injuries",
  "Earthquake severe damage to Kathmandu. Tragic loss of life.",
  "7.9-magnitude earthquake strikes Nepal, damage reported",
  "Thoughts are with the families in #Nepal")

str_detect(string = tweets, pattern = "magnitude|damage")

## [1]  TRUE  TRUE  TRUE FALSE

This may be useful for alternative spellings.

str_detect(string = c("labor", "labour", "workforce"),
           pattern = "labor|labour")

## [1]  TRUE  TRUE FALSE

1.1.4 Character Classes

Character classes are lists of characters that belong to a group, such as alphabetic, numeric, alphanumeric characters, etc. It is possible to build them or to use predefined classes. They are written by placing them in square brackets []. For example, if the aim is to match strings where characters

Let us assume that we face file names with a date, and that we want to match only those whose month in a given year is “January” or “February”:

str_extract(string = c("file_2019-01-01.txt", "file_2019-03-01.txt", "file_2019-02-01.txt"),
            pattern = c("file_2019-0[12]-01"))

## [1] "file_2019-01-01" NA                "file_2019-02-01"

In the previous code, we therefore searched each string for the occurrence of the substring file_2019-01-01 or file_2019-02-01.

Using a dash -, it is possible to define a sequence of characters. Thus, the character class [A-Z] is used to match the letters of the following set: ABCDEFGHIJKLMNOPQRSTUVWXYZ. The character class [0-9] mathces the character set 0123456789.

str_extract(string = c("file_2019-01-01.txt", "file_2019-03-01.txt", "file_2019-02-01.txt"),
            pattern = c("file_2019-0[1-3]-01"))

## [1] "file_2019-01-01" "file_2019-03-01" "file_2019-02-01"

Unions of groups can be made:

str_extract(string = c("file_2019-01-01.txt", "file_2019-02-01.txt",
                       "file_2019-03-01.txt", "file_2019-04-01.txt",
                       "file_2019-05-01.txt", "file_2019-06-01.txt",
                       "file_2019-07-01.txt", "file_2019-08-01.txt"
                       ),
            pattern = c("file_2019-0[1-36-8]-01"))

## [1] "file_2019-01-01" "file_2019-02-01" "file_2019-03-01" NA               
## [5] NA                "file_2019-06-01" "file_2019-07-01" "file_2019-08-01"

To exclude a group of characters, a circumflex accent ^ can be used:

str_extract(string = c("file_2019-01-01.txt", "file_2019-02-01.txt",
                       "file_2019-03-01.txt", "file_2019-04-01.txt",
                       "file_2019-05-01.txt", "file_2019-06-01.txt",
                       "file_2019-07-01.txt", "file_2019-08-01.txt"
                       ),
            pattern = c("file_2019-0[1-36-8]-01"))

## [1] "file_2019-01-01" "file_2019-02-01" "file_2019-03-01" NA               
## [5] NA                "file_2019-06-01" "file_2019-07-01" "file_2019-08-01"

If, on the other hand, the circumflex must be part of the character class, it should not be placed right after the opening bracket:

str_extract(string = c("So happy ^_^", "So happy (#^.^#)"),
            pattern = "happy [a-z^]")

## [1] "happy ^" NA

It is also possible to escape the character as follows: [\^].

Somes classes are pre-built and can be referred to by their name. They are based on the POSIX family of standards. The most used (in my own experience) are listed in the Table below.

To refer to these classes, they need to be put between the brackets defining the character classes:

tweets

## [1] "PANIC IN NEPAL: Strong quake hits capital, causing major damage, injuries"
## [2] "Earthquake severe damage to Kathmandu. Tragic loss of life."              
## [3] "7.9-magnitude earthquake strikes Nepal, damage reported"                  
## [4] "Thoughts are with the families in #Nepal"

str_detect(tweets, "magnitude [[:digit:]]")

## [1] FALSE FALSE FALSE FALSE

Some classes also benefit from an abbreviation:

str_extract_all("Magnitude 6.1", "\\d")

## [[1]]
## [1] "6" "1"

str_extract_all("Magnitude 6.1", "\\D")

## [[1]]
##  [1] "M" "a" "g" "n" "i" "t" "u" "d" "e" " " "."

str_extract_all("Magnitude 6.1", "\\w")

## [[1]]
##  [1] "M" "a" "g" "n" "i" "t" "u" "d" "e" "6" "1"

str_extract_all("Magnitude 6.1", "\\W")

## [[1]]
## [1] " " "."

1.1.5 Grouping

Parentheses can be used to group some part of a regular expression together. This is particularly helpful when combined with quantifiers and character classes, to manipulate file names for example.

Here is an example with the function str_extract(), which extracts matching patterns from a string:

str_extract(string = c("to analyse", "to analyze", "other"),
           pattern = "analy(s|z)e")

## [1] "analyse" "analyze" NA

1.1.6 Quantifiers

Quantifiers are used to repeat the regular expression a given number of times. The Table below lists the available quantifiers. They are placed after the regex that need to be matched a given number of times.

str_extract(string = c("The labour force", "The labor force"),
           pattern = "labou?r")

## [1] "labour" "labor"

Combining quantifiers with character classes or groups allows to match more complex patterns:

tweets

## [1] "PANIC IN NEPAL: Strong quake hits capital, causing major damage, injuries"
## [2] "Earthquake severe damage to Kathmandu. Tragic loss of life."              
## [3] "7.9-magnitude earthquake strikes Nepal, damage reported"                  
## [4] "Thoughts are with the families in #Nepal"

str_extract(tweets, "magnitude [[:digit:]\\.]{1,}")

## [1] NA NA NA NA

str_extract_all(tweets, "\\w+")

## [[1]]
##  [1] "PANIC"    "IN"       "NEPAL"    "Strong"   "quake"    "hits"    
##  [7] "capital"  "causing"  "major"    "damage"   "injuries"
## 
## [[2]]
## [1] "Earthquake" "severe"     "damage"     "to"         "Kathmandu" 
## [6] "Tragic"     "loss"       "of"         "life"      
## 
## [[3]]
## [1] "7"          "9"          "magnitude"  "earthquake" "strikes"   
## [6] "Nepal"      "damage"     "reported"  
## 
## [[4]]
## [1] "Thoughts" "are"      "with"     "the"      "families" "in"      
## [7] "Nepal"

(.*) is a useful combination of grouping and quantifiers. It allows to match any sequence of characters:

• .: any character
• *: present zero or more times

x <- c("type_1_20190101_20190131.txt", "type_2_20190101_20190131.txt",
       "type_1_20190201_20190228.txt", "type_2_20190201_20190228.txt",
       "type_1_20190101_20190131.csv", "type_2_20190101_20190131.csv",
       "type_1_20190201_20190228.csv", "type_2_20190201_20190228.csv")
str_extract(x, "^type_1(.*)\\.txt$")

## [1] "type_1_20190101_20190131.txt" NA                            
## [3] "type_1_20190201_20190228.txt" NA                            
## [5] NA                             NA                            
## [7] NA                             NA

1.1.7 Some functions

To illustrate the examples of regular expressions, we used some functions of the package {stringr} which all begin with the prefix str_.

# French phone numbers
phone_numbers <-
  c("02 23 23 35 45", "02-23-23-35-45", 
    "Madrid", "02.23.23.35.45", "0223233545",
    "Milan", "02 23 23 35 45  ",
    " 02 23 23 35 45", "Home: 02 23 23 35 45")

pattern_phone_number <- str_c(str_dup("([0-9]{2})[- \\.]", 4),  "([0-9]{2})")
pattern_phone_number

## [1] "([0-9]{2})[- \\.]([0-9]{2})[- \\.]([0-9]{2})[- \\.]([0-9]{2})[- \\.]([0-9]{2})"

# Extract phone numbers
str_extract(phone_numbers, pattern_phone_number)

## [1] "02 23 23 35 45" "02-23-23-35-45" NA               "02.23.23.35.45"
## [5] NA               NA               "02 23 23 35 45" "02 23 23 35 45"
## [9] "02 23 23 35 45"

# Extract phone numbers, then remove punctuation and white characters
str_extract(phone_numbers, pattern_phone_number) %>% 
  str_replace_all("[[:punct:]\\s]", "")

## [1] "0223233545" "0223233545" NA           "0223233545" NA          
## [6] NA           "0223233545" "0223233545" "0223233545"

# Extract matched groups from the phone numbers
str_match(phone_numbers, pattern_phone_number)

##       [,1]             [,2] [,3] [,4] [,5] [,6]
##  [1,] "02 23 23 35 45" "02" "23" "23" "35" "45"
##  [2,] "02-23-23-35-45" "02" "23" "23" "35" "45"
##  [3,] NA               NA   NA   NA   NA   NA  
##  [4,] "02.23.23.35.45" "02" "23" "23" "35" "45"
##  [5,] NA               NA   NA   NA   NA   NA  
##  [6,] NA               NA   NA   NA   NA   NA  
##  [7,] "02 23 23 35 45" "02" "23" "23" "35" "45"
##  [8,] "02 23 23 35 45" "02" "23" "23" "35" "45"
##  [9,] "02 23 23 35 45" "02" "23" "23" "35" "45"

The str_locate() and str_locate_all() functions return the start and end indices of the matched subchains.

tweets

## [1] "PANIC IN NEPAL: Strong quake hits capital, causing major damage, injuries"
## [2] "Earthquake severe damage to Kathmandu. Tragic loss of life."              
## [3] "7.9-magnitude earthquake strikes Nepal, damage reported"                  
## [4] "Thoughts are with the families in #Nepal"

str_locate(string = tweets, pattern = "magnitude")

##      start end
## [1,]    NA  NA
## [2,]    NA  NA
## [3,]     5  13
## [4,]    NA  NA

str_locate_all(string = tweets, pattern = "magnitude")

## [[1]]
##      start end
## 
## [[2]]
##      start end
## 
## [[3]]
##      start end
## [1,]     5  13
## 
## [[4]]
##      start end

To look for a pattern in a string ignoring case sensitivity, the pattern can be previously provided to the function stringr::regex():

s <- c("earthquake", "Earthquake")
str_detect(string = s, pattern = "earthquake")

## [1]  TRUE FALSE

str_detect(string = s,
           pattern = stringr::regex("earthquake", ignore_case = TRUE))

## [1] TRUE TRUE

2.2.1 Some functions to help clean texts

To clean the tweets, we will create some functions. Let us define the function remove_url() to remove URLs from a string, using a regex :

#' remove_url
#' Removes URLs from a string
#' @param x string
remove_url <- function(x){
  pattern_url <- "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
  str_replace_all(string = x, pattern = pattern_url, replacement = "")
}

Let us also create the function remove_special_chars() allowing to remove special characters found in tweets (unrecognized apostrophes) :

#' remove_special_chars
#' Removes the special characters from a string
#' @param x string
remove_special_chars <- function(x){
  str_replace_all(string = x,
                  pattern = "[^\x20-\x7e]", replacement = "")
}

Let us define the function remove_mentions() which identifies a mention in a Twitter status (starting with the arobase symbol (@) and followed by the user’s screen name) :

#' remove_mentions
#' Removes the mentions from tweets (`@`)
#' @param x string
remove_mentions <- function(x){
  str_replace_all(x, "@[[:alnum:]]+\\s?", "")
}

We can also define the functions remove_punctuation() and remove_numbers() which remove punctuation and numbers, respectively, from a tweet.

#' remove_punctuation
#' Removes punctuation from tweets
#' @param x string
remove_punctuation <- function(x){
  str_replace_all(x, "[[:punct:]]", "")
}

#' remove_numbers
#' Removes numbers from tweets
#' @param x string
remove_numbers <- function(x){
  str_replace_all(x, "[[:digit:]]", "")
}

Eventuallt, we can define the function remove_char_ref() which removes character references (e.g., & )

#' remove_char_ref
#' Removes character reference
#' @param x string
remove_char_ref <- function(x){
  str_replace_all(x, "&[[:alpha:]]{1,6};", "")
}

2.2.2 Building the Corpus

We will create a corpus of texts from the tweets. The idea is to obtain an object containing as many documents as statuses. For each document, we have to count the occurrence of each of the words encountered throughout the corpus. At the end of the day, we obtain a matrix whose rows correspond to the tweets and whose columns indicate the occurrence of each word. The columns of this matrix will be the explanatory variables that can be uses to train a classifier.

First, let us create a tibble with an identifier and a tweet.

tweets_earthquake_tt <- 
  tibble(id = 1:nrow(tweets_earthquake),
         text = tweets_earthquake$full_text)
tweets_earthquake_tt

## # A tibble: 7,252 x 2
##       id text                                                              
##    <int> <chr>                                                             
##  1     1 "Dua's for all those affected by the\nearthquakes in India,Nepal …
##  2     2 Absolutely devastated by the destruction to my old home #Nepal    
##  3     3 Thoughts are with the families in #Nepal                          
##  4     4 Frightful images! Our prayers echo for everyone affected. #earthq…
##  5     5 Who was Gajendra Singh ? Today no news boz , of earthquake in Nep…
##  6     6 Live: Nepal cabinet meets to seek foreign help, 114 feared dead a…
##  7     7 When you go out for Momos this evening, ask and reassure the sell…
##  8     8 A crucial tool in a situation like #NepalQuake #NepalEarthquake..…
##  9     9 our affection from Madrid Spain, there we were this summer from N…
## 10    10 Devastating pictures of #NepalEarthQuake http://t.co/VaEOUkUTsG   
## # … with 7,242 more rows

Then, let us apply to each tweet the functions to clean the text:

tweets_earthquake_tt <- 
  tweets_earthquake_tt %>% 
  mutate(cleaned_text = str_to_lower(text),
         cleaned_text = remove_url(cleaned_text),
         cleaned_text = remove_mentions(cleaned_text),
         cleaned_text = remove_char_ref(cleaned_text),
         cleaned_text = remove_special_chars(cleaned_text),
         cleaned_text = remove_punctuation(cleaned_text),
         cleaned_text = remove_numbers(cleaned_text)
  )
tweets_earthquake_tt

## # A tibble: 7,252 x 3
##       id text                            cleaned_text                      
##    <int> <chr>                           <chr>                             
##  1     1 "Dua's for all those affected … "duas for all those affected by t…
##  2     2 Absolutely devastated by the d… absolutely devastated by the dest…
##  3     3 Thoughts are with the families… thoughts are with the families in…
##  4     4 Frightful images! Our prayers … "frightful images our prayers ech…
##  5     5 Who was Gajendra Singh ? Today… who was gajendra singh  today no …
##  6     6 Live: Nepal cabinet meets to s… "live nepal cabinet meets to seek…
##  7     7 When you go out for Momos this… when you go out for momos this ev…
##  8     8 A crucial tool in a situation … "a crucial tool in a situation li…
##  9     9 our affection from Madrid Spai… our affection from madrid spain t…
## 10    10 Devastating pictures of #Nepal… "devastating pictures of nepalear…
## # … with 7,242 more rows

Using the unnest_tokens() function of {tidytext}, let us separate each word from the tweets. Each line of the tibble obtained is a tuple indicating the identifier and the word.

tweets_earthquake_tt <- 
  tweets_earthquake_tt %>% 
  select(id, cleaned_text) %>% 
  unnest_tokens(word, cleaned_text)

tweets_earthquake_tt

## # A tibble: 82,692 x 2
##       id word          
##    <int> <chr>         
##  1     1 duas          
##  2     1 for           
##  3     1 all           
##  4     1 those         
##  5     1 affected      
##  6     1 by            
##  7     1 theearthquakes
##  8     1 in            
##  9     1 indianepal    
## 10     1 bhutan        
## # … with 82,682 more rows

Some very frequent and potentially noisy words can be removed. A list of such words is available in the tibble stop_words. The function get_stopwords() can also be used

# English stop words
stop_words

## # A tibble: 1,149 x 2
##    word        lexicon
##    <chr>       <chr>  
##  1 a           SMART  
##  2 a's         SMART  
##  3 able        SMART  
##  4 about       SMART  
##  5 above       SMART  
##  6 according   SMART  
##  7 accordingly SMART  
##  8 across      SMART  
##  9 actually    SMART  
## 10 after       SMART  
## # … with 1,139 more rows

# French stop words
stop_words_fr <- get_stopwords(language = "fr")
stop_words_fr

## # A tibble: 164 x 2
##    word  lexicon 
##    <chr> <chr>   
##  1 au    snowball
##  2 aux   snowball
##  3 avec  snowball
##  4 ce    snowball
##  5 ces   snowball
##  6 dans  snowball
##  7 de    snowball
##  8 des   snowball
##  9 du    snowball
## 10 elle  snowball
## # … with 154 more rows

To remove these stopwords, we can use the anti_joint() function from {dplyr}:

tweets_earthquake_tt <- 
  tweets_earthquake_tt %>% 
  anti_join(stop_words, by = c("word"))
tweets_earthquake_tt

## # A tibble: 44,011 x 2
##       id word          
##    <int> <chr>         
##  1     1 duas          
##  2     1 affected      
##  3     1 theearthquakes
##  4     1 indianepal    
##  5     1 bhutan        
##  6     1 staysafe      
##  7     1 form          
##  8     1 equake        
##  9     2 absolutely    
## 10     2 devastated    
## # … with 44,001 more rows

To extract the root of each word the wordStem() function from {SnowballC} can be applied:

tweets_earthquake_tt <- 
  tweets_earthquake_tt %>% 
  mutate(word_stem = wordStem(word))
tweets_earthquake_tt

## # A tibble: 44,011 x 3
##       id word           word_stem   
##    <int> <chr>          <chr>       
##  1     1 duas           dua         
##  2     1 affected       affect      
##  3     1 theearthquakes theearthquak
##  4     1 indianepal     indianep    
##  5     1 bhutan         bhutan      
##  6     1 staysafe       staysaf     
##  7     1 form           form        
##  8     1 equake         equak       
##  9     2 absolutely     absolut     
## 10     2 devastated     devast      
## # … with 44,001 more rows

Using the count() function of {dplyr}, the occurrence of each word is easily calculated:

freq_words <- 
  tweets_earthquake_tt %>% 
  dplyr::count(word_stem, sort = TRUE)
freq_words

## # A tibble: 8,375 x 2
##    word_stem          n
##    <chr>          <int>
##  1 nepal           3820
##  2 nepalearthquak  1164
##  3 earthquak        670
##  4 prayer           486
##  5 peopl            457
##  6 god              455
##  7 nepalquak        370
##  8 donat            285
##  9 new              274
## 10 india            251
## # … with 8,365 more rows

We can use a barplot to graph the occurrences of the top n words (here, we use n=10):

freq_words %>% slice(1:10) %>% 
  ggplot(data = .,
         aes(x = reorder(word_stem, n), y = n)) +
  geom_bar(stat ="identity") +
  labs(x = "Word", y = "Frequency") +
  coord_flip()

A word cloud can be drawn using the wordcloud() function from {wordcloud}. The size of words is positively related to the frequency of appearance.

library(wordcloud)
wordcloud(words = freq_words %>% slice(1:100) %>% 
            magrittr::extract2("word_stem"),
          freq = freq_words %>% slice(1:100) %>%
            magrittr::extract2("n"),
          random.order = F)