| Expect for Windows® User Guide |

# Regular Expressions Primer

- Regular Expressions Primer Overview
- - About Regular Expressions
  - About Regex Syntax
  - Building Simple Patterns
  - - Matching Simple Strings
    - Searching with Wildcards
    - Searching for Special Characters
    - Ranges and Repetition
    - - Ranges, {min, max}
      - Repetition, ?*+
      - Quantifier Summary
    - Using Conditional Expressions
    - Grouping Similar Items in Parentheses
    - Matching Sequences
    - - Building Simple Character Classes
      - Preventing Matches with Character Classes
      - Compound Character Classes
      - Character Class Summary
    - Matching Locations within a Sting
  - More Tcl Regex Resources

## Regular Expressions Primer Overview

The Regular Expressions Primer is a tutorial for those completely new to regular expressions. To familiarize you with regular expressions, this primer starts with the simple building blocks of the syntax and through examples, builds to construct complex expressions.

Regular expressions are embedded in programs to parse text. For example, an Expect Tcl script might contain the following string to anonymously log in to an FTP server:

      expect -re "([Ll]ogin|Username):.*" {
          send "anonymous\r"
      }

The `([Ll]ogin|Username)` regular expression matches the following strings: "Login", "login", and "Username". When one of these strings is matched, Expect sends the string "anonymous" to the FTP server. Note that the `expect -re` command must prefix all Tcl regular expressions. Without `expect -re`, the command defaults to matching glob patterns. In glob pattern matching, the wildcards are "*" (which matches any sequence of characters) and "?" (which matches a single character). A literal "*", "?" or "\" might be matched by escaping with a backslash. See the Tcl Glob Manual Page for more information.

## About Regular Expressions

Regular expressions are used to describe patterns of characters that match against text strings. They can be used as a tool to search for and replace text, manipulate data, or test for a certain condition in a string of characters. Many everyday tasks can be accomplished with regular expressions, such as checking for the occurrence of a specific word or phrase in the body of an e-mail message, or finding specific file types, such as `.txt` files, in a folder or directory. Regular expressions are often called "regex", "regexes", "regexps", and "RE". This primer uses the terms "regular expressions", "regex", and "regexes" equally.

## About Regex Syntax

Regular expressions use syntax elements comprised of alphanumeric characters and symbols. For example, the regex `(2)` searches for the number 2, while the regex `([1-9][0-9]{2}-[0-9]{4})` matches a regular 7-digit phone number.

There are many flavors and types of regular expression syntax. These variations are found in various tools, languages and operating systems. For example, Tcl, Perl, Python, grep, sed, VI, and Unix all use variations on standard regex syntax. This primer focuses on standard regex patterns not tied to a specific language or tool. This standard syntax can be later applied to the specific language, tool or application of your choice.

## Building Simple Patterns

Complete regular expressions are constructed using characters as small building block units. Each building block is in itself simple, but since these units can be combined in an infinite number of ways, knowing how to combine them to achieve a goal takes some practice. This section shows you how to build regexes through examples ranging from the simple to the more complex.

### Matching Simple Strings

The simplest and most common type of regex is an alphanumeric string that matches itself, called a "literal text match". A literal text regex matches anywhere along a string. For example, a literal string matches itself when placed alone, and at the beginning, middle, or end of a larger string. Literal text matches are case sensitive.

Using regexes to search for simple strings.

Example 1: Search for the string "at".

- Regex:

          at

- Matches:

          at
          math
          hat
          ate

- Doesn't Match:

          it
          a-t
          At

Example 2: Search for the string "email".

- Regex:

          email

- Matches:

          email
          emailing
          many_emails

- Doesn't Match:

          Email
          EMAILing
          e-mails

Example 3: Search for the alphanumeric string "abcdE567".

- Regex:

          abcdE567

- Matches:

          abcdE567
          AabcdE567ing
          text_abcdE567

- Doesn't Match:

          SPAMabCdE567
          ABCDe567

Note: Regular expressions are case sensitive unless case is deliberately modified.

### Searching with Wildcards

In the previous examples, regular expressions are constructed with literal characters that match themselves. There are other characters in regex syntax that match in a more generalized way. These are called "metacharacters". Metacharacters do not match themselves, but rather perform a specific task when used in a regular expression. One such metacharacter is the dot ".", or wildcard. When used in a regular expression, the wildcard can match any single character.

Using the wildcard to match any character.

Example 1: Use a wildcard to search for any one character before the string "ubject:".

- Regex:

          .ubject:

- Matches:

          Subject:
          subject:
          Fubject:

- Doesn't Match:

          Subject
          subject

Example 2: Use three dots "`...`" to search for any three characters within a string.

- Regex:

          t...s

- Matches:

          trees
          tEENs
          t345s
          t-4-s

- Doesn't Match:

         Trees
         twentys
         t1234s

Example 3: Use several wildcards to match characters throughout a string.

- Regex:

          .a.a.a

- Matches:

          Canada
          alabama
          banana
          3a4a5a

- Doesn't Match:

          aaa

### Searching for Special Characters

In regular expression syntax, most non-alphanumerical characters are treated as special characters. These characters, called "metacharacters", include asterisks, question marks, dots, slashes, etc. In order to search for a metacharacter without using its special attribute, precede it with a backslash "\" to change it into a literal character. For example, to build a regex to search for a `.txt` file, precede the dot with a backslash `\.txt` to prevent the dot's special function, a wildcard search. The backslash, called an "escape character" in regex terminology, turns metacharacters into literal characters.

Precede the following metacharacters with a backslash "\" to search for them as literal characters:

`^ $ + * ? . | ( ) { } [ ] \`

Using the backslash "\" to escape special characters in a regular expression.

Example 1: Escape the dollar sign "$" to find the alphanumeric string "$100".

- Regex:

          \$100

- Matches:

          $100
          $1000

- Doesn't Match:

          \$100
          100

Example 2: Use the dot "." as a literal character to find a file called "email.txt".

- Regex:

          email\.txt

- Matches:

          email.txt

- Doesn't Match:

          email
          txt
          email_txt

Example 3: Escape the backslash "\" character to search for a Windows file.

- Regex:

          c:\\readme\.txt

- Matches:

          c:\readme.txt

- Doesn't Match:

          c:\\readme.txt
          d:\readme.txt
          c:/readme.txt

### Ranges and Repetition

Regex syntax includes metacharacters which specify the number of times a particular character or string must match. This group of metacharacters are called "quantifiers"; they influence the quantity of matches found. Quantifiers act on the element immediately preceding them, which could be a digit, a letter, or another metacharacter (including spaces as metacharacters not previously defined and the dot "."). This section demonstrates how quantifers search using ranges and repetition.

#### Ranges, {min, max}

Ranges are considered "counting qualifiers" in regular expressions. This is because they specify the minimum number of matches to find and the maximum number of matches to allow. Use ranges in regex searches when a bound, or a limit, should be placed on search results. For example, the range `{3,5}` matches an item at least 3 times, but not more than 5 times. When this range is combined with the regex, `a{3,5}`, the strings "aaa", "aaaa", and "aaaaa" are successfully matched. If only a single number is expressed within curly braces `{3}`, the pattern matches exactly three items. For example, the regex `b{3}` matches the string "bbb".

Using ranges to identify search patterns.

Example 1: Match the preceding "0" at least 3 times with a maximum of 5 times.

- Regex:

          60{3,5} years

- Matches:

          6000 years
          60000 years
          600000 years

- Doesn't Match:

          60 years
          600 years
          6003 years
          6000000 years

Example 2: Using the "." wildcard to match any character sequence two or three characters long.

- Regex:

          .{2,3}

- Matches:

          404
          44
          com
          w3

- Doesn't Match:

          4
          a
          aaaa

Example 3: Match the preceding "e" exactly twice.

- Regex:

          be{2}t

- Matches:

          beet

- Doesn't Match:

          bet
          beat
          eee

Example 4: Match the preceding "w" exactly three times.

- Regex:

          w{3}\.mydomain\.com

- Matches:

          www.mydomain.com

- Doesn't Match:

          web.mydomain.com
          w3.mydomain.com

#### Repetition, ?*+

Unlike range quantifiers, the repetition quantifiers (question mark "?", asterisk "*", and plus "+") have few limits when performing regex searches, they are greedy. This is significant because these quantifiers settle for the minimum number of required matches, but always attempt to match as many times as possible, up to the maximum allowed. For example, the question mark "?" matches any preceding character 0 or 1 times, the asterisk "*" matches the preceding character 0 or more times, and the plus "+" matches the preceding character 1 or more times. Use repetition quantifiers in regex searches when large numbers of results are desired.

Using repetition to search for repeated characters with few limits.

Example 1: Use "?" to match the "u" character 0 or 1 times.

- Regex:

          colou?r

- Matches:

          colour
          color

- Doesn't Match:

          colouur
          Colour

Example 2: Use "*" to match the preceding item 0 or more times; use "." to match any character.

- Regex:

          www\.my.*\.com

- Matches:

          www.mysite.com
          www.mypage.com
          www.my.com

- Doesn't Match:

          www.oursite.com
          mypage.com

Example 3: Use "+" to match the preceding "5" at least once.

- Regex:

          bob5+@foo\.com

- Matches:

          bob5@foo.com
          bob5555@foo.com

- Doesn't Match:

          bob@foo.com
          bob65555@foo.com

#### Quantifier Summary

The following table defines the various regex quantifiers. Note that each quantifier is unique and will perform a varying minimum and maximum number of matches in order to search successfully.

Quantifier | Description
{num} | Matches the preceding element num times.
{min, max} | Matches the preceding element at least min times, but not more than max times.
? | Matches any preceding element 0 or 1 times.
* | Matches the preceding element 0 or more times.
+ | Matches the preceding element 1 or more times.

### Using Conditional Expressions

Conditional expressions help qualify and restrict regex searches, increasing the probability of a desirable match. The vertical bar "|" symbol, meaning "OR", places a condition on the regex to search for either one character in a string or another. Because the regex has a list of alternate choices to evaluate, this regex technique is called "alternation". To search for either one character or another, insert a vertical bar "|" between the desired characters.

Example 1: Use "|" to alternate a search for various spellings of a string.

- Regex:

          gray|grey

- Matches:

          gray
          grey

- Doesn't Match:

          GREY
          Gray

Example 2: Use "|" to alternate a search for either email or Email or EMAIL or e-mail.

- Regex:

          email|Email|EMAIL|e-mail

- Matches:

          email
          Email
          EMAIL
          e-mail

- Doesn't Match:

          EmAiL
          E-Mail

### Grouping Similar Items in Parentheses

Use parentheses to enclose a group of related search elements. Parentheses limit scope on alternation and create substrings to enhance searches with metacharacters. For example, use parentheses to group the expression `(abc)`, then apply the range quantifier `{3}` to find instances of the string "abcabcabc".

Along with grouping expressions into subpatterns, parentheses also capture each part of a matched string. It should be noted that this further parsing requires additional time and resources to complete.

Using parentheses to group regular expressions.

Example 1: Use parentheses and a range quantifier to find instances of the string "abcabcabc".

- Regex:

          (abc){3}

- Matches:

          abcabcabc
          abcabcabcabc

- Doesn't Match:

          abc
          abcabc

Example 2: Use parentheses to limit the scope of alternative matches on the words gray and grey.

- Regex:

          gr(a|e)y

- Matches:

          gray
          grey

- Doesn't Match:

          gry
          graey

Example 3: Use parentheses and "|" to locate past correspondence in a mail-filtering program. This regex finds a 'To:' or a 'From:' line followed by a space and then either the word 'Smith' or the word 'Chan'.

- Regex:

          (To:|From:)(Smith|Chan)

- Matches:

          To:Smith
          To:Chan
          From:Smith
          To:Smith, Chan
          To:Smithe
          From:Channel4News

- Doesn't Match:

          To:smith
          To:All
          To:Schmidt

### Matching Sequences

You can build a regular expression to match a sequence of characters. These sequences, called "character classes", simply place a set of characters side-by-side within square brackets "[]". An item in a character class can be either an ordinary character, representing itself, or a metacharacter, performing a special function. This primer covers how to build simple character classes, prevent matches with character classes, and construct compound character classes with metacharacters.

#### Building Simple Character Classes

The most basic type of character class is a set of characters placed side-by-side within square brackets "[]". For example, the regular expression `[bcr]at`, matches the words "bat", "cat", or "rat" because it uses a character class (that includes "b","c", or "r") as its first character. Character classes only match singular characters unless a quantifier is placed after the closing bracket. For examples using quantifiers with character classes, see Compound Character Classes. The following table shows how to use simple character classes in regex searches.

Note: When placed inside a character class, the hyphen "-" metacharacter denotes a continuious sequence of letters or numbers in a range. For example, [a-d] is a range of letters denoting the continuious sequence of a,b,c and d. When a hyphen is otherwise used in a regex, it matches a literal hyphen.

Using simple character classes to perform regex searches.

Example 1: Use a character class to match all cases of the letter "s".

- Regex:

          Java[Ss]cript

- Matches:

          JavaScript
          Javascript

- Doesn't Match:

          javascript
          javaScript

Example 2: Use a character class to limit the scope of alternative matches on the words gray and grey.

- Regex:

          gr[ae]y

- Matches:

          gray
          grey

- Doesn't Match:

          gry
          graey

Example 3: Use a character class to match any one digit in the list.

- Regex:

          [0123456789]

- Matches:

          5
          0
          9

- Doesn't Match:

          x
          ?
          F

Example 4: To simplify the previous example, use a hyphen "-" within a character class to denote a range for matching any one digit in the list.

- Regex:

          [0-9]

- Matches:

          5
          0
          9

- Doesn't Match:

          234
          42

Example 5: Use a hyphen "-" within a character class to denote an alphabetic range for matching various words ending in "mail".

- Regex:

          [A-Z]mail

- Matches:

          Email
          Xmail
          Zmail

- Doesn't Match:

          email
          mail

Example 6: Match any three or more digits listed in the character class.

- Regex:

          [0-9]{3,}

- Matches:

          012
          1234
          555
          98754378623

- Doesn't Match:

          10
          7

#### Preventing Matches with Character Classes

Previous examples used character classes to specify exact sequences to match. Character classes can also be used to prevent, or negate, matches with undesirable strings. To prevent a match, use a leading caret "^" (meaning NOT), within square brackets, `[^...]`. For example, the regex `[^a]` matches any single character except the letter "a".

Note: The caret symbol must be the first character within the square brackets to negate a character class.

Using character classes to prevent a sequence from matching.

Example 1: Prevent a match on any numeric string. Use the "*" to match an item 0 or more times.

- Regex:

          [^0-9]*

- Matches:

          abc
          c
          Mail
          u-see
          a4a

- Doesn't Match:

          1
          42
          100
          23000000

Example 2: Search for a text file beginning with any character not a lower-case letter.

- Regex:

          [^a-z]\.txt

- Matches:

          A.txt
          4.txt
          Z.txt

- Doesn't Match:

          r.txt
          a.txt
          Aa.txt

Example 3: Prevent a match on the numbers "10" and "12".

- Regex:

          1[^02]

- Matches:

          13
          11
          19
          17
          1a

- Doesn't Match:

          10
          12
          42
          a1

#### Compound Character Classes

Character classes are a versatile tool when combined with various pieces of the regex syntax. Compound character classes can help clarify and define sophisticated searches, test for certain conditions in a program, and filter wanted e-mail from spam. This section uses compound character classes to build meaningful expressions with the regex syntax.

Using compound character classes with the regex syntax.

Example 1: Find a partial e-mail address. Use a character class to denote a match for any number between 0 and 9. Use a range to restrict the number of times a digit matches.

- Regex:

          smith[0-9]{2}@

- Matches:

          smith44@
          smith42@

- Doesn't Match:

          Smith34
          smith6
          Smith0a

Example 2: Search an HTML file to find each instance of a header tag. Allow matches on whitespace after the tag but before the ">".

- Regex:

          (<[Hh][1-6] *>)

- Matches:

           <H1>
           <h6>
           <H3  >
           <h2    >

- Doesn't Match:

          <H1
          <   h2>
          <a1>

Example 3: Match a regular 7-digit phone number. Prevent the digit "0" from leading the string.

- Regex:

          ([1-9][0-9]{2}-[0-9]{4})

- Matches:

           555-5555
           123-4567

- Doesn't Match:

          555.5555
          1234-567
          023-1234

Example 4: Match a valid web-based protocol. Escape the two front slashes.

- Regex:

          [a-z]+:\/\/

- Matches:

          http://
          ftp://
          tcl://
          https://

- Doesn't Match:

          http
          http:
          1a3://

Example 5: Match a valid e-mail address.

- Regex:

          [a-z0-9_-]+(\.[a-z0-9_-]+)*@[a-z0-9_-]+(\.[a-z0-9_-]+)+

- Matches:

          j_smith@foo.com
          j.smith@bc.canada.ca
          smith99@foo.co.uk
          1234@mydomain.net

- Doesn't Match:

          @foo.com
          .smith@foo.net
          smith.@foo.org
          www.myemail.com

#### Character Class Summary

The following table defines various character class sequences. Use these alphanumeric patterns to simplify your regex searches.

Character Class | Description
`[0-9]` | Matches any digit from 0 to 9.
`[a-zA-z]` | Matches any alphabetic character.
`[a-zA-z0-9]` | Matches any alphanumeric character.
`[^0-9]` | Matches any non-digit.
`[^a-zA-z]` | Matches any non-alphabetic character.

### Matching Locations within a String

At times, the pattern to be matched appears at either the very beginning or end of a string. In these cases, use a caret "^" to match a desired pattern at the beginning of a string, and a dollar sign "$" for the end of the string. For example, the regular expression `email` matches anywhere along the following strings: "email", "emailing", "bogus_emails", and "smithsemailaddress". However, the regex `^email` only matches the strings "email" and "emailing". The caret "^" in this example is used to effectively anchor the match to the start of the string. For this reason, both the caret "^" and dollar sign "$" are referred to as anchors in the regex syntax.

Note: The caret "^" has many meanings in regular expressions. Its function is determined by its context. The caret can be used as an anchor to match patterns at the beginning of a string, for example:`(^File)`. The caret can also be used as a logical "NOT" to negate content in a character class, for example: `[^...]`.

Using anchors to match at the beginning or end of a string.

Example 1: Use "$" to match the ".com" pattern at the end of a string.

- Regex:

          .*\.com$

- Matches:

          mydomain.com
          a.b.c.com

- Doesn't Match:

          mydomain.org
          mydomain.com.org

Example 2: Use "^" to match "inter" at the beginning of a string, "$" to match "ion" at the end of a string, and ".*" to match any number of characters within the string.

- Regex:

          ^inter.*ion$

- Matches:

          internationalization
          internalization

- Doesn't Match:

          reinternationalization

Example 3: Use "^" inside parentheses to match "To" and "From" at the beginning of the string.

- Regex:

          (^To:|^From:)(Smith|Chan)

- Matches:

          From:Chan
          To:Smith
          From:Smith
          To:Chan

- Doesn't Match:

          From: Chan
          from:Smith
          To Chan

Example 4: Performing the same search as #3, place the caret "^" outside the parentheses this time for similar results.

- Regex:

          ^(From|Subject|Date):(Smith|Chan|Today)

- Matches:

          From:Smith
          Subject:Chan
          Date:Today

- Doesn't Match:

          X-Subject:
          date:Today

## More Tcl Regex Resources

### ActiveState Programmer Network (ASPN)

- re_syntax, Tcl regular expressions syntax
- regexp, match a regular expression against a string
- regsub, perform substitutions based on regular expression pattern matching

### Tcl Developer Xchange

- Regular Expressions in Tcl 8.1
- Evaluating Scripts with Switch

### O'Reilly Books

- Exploring Expect, "Chapter 5: Regular Expressions", by Don Libes
- Mastering Regular Expressions, by Jeffrey Friedl