Trustwave Knowledge Base

KB Home Search Latest Additions Most Popular

Knowledgebase

Home » Knowledgebase » Trustwave MailMarshal (SEG) » HOWTO: How do I use Header Rewrite / Regular Expressions?

HOWTO: How do I use Header Rewrite / Regular Expressions?

Show/Hide Article Tools

This article applies to:

Trustwave MailMarshal (SEG)
Trustwave ECM/MailMarshal Exchange
Trustwave SPE (MailMarshal SPE)

Question:

How do I use Header Rewrite / Regular Expressions?

Reply:

The Header Rewrite function is an advanced tool which allows email addressing and envelope information to be modified by Rules. (In SEG, Header Rewrite can also be applied globally before Rules are applied). This article provides detailed information on the syntax of the Regular Expression substitution engine used by Header Rewrite.

Note: What follows is the reference material for the Regular Expression engine actually used in the products. It is authoritative but not particularly easy to read. Additional resources for general information about Regular Expressions may be found on the Web. At this writing one available resource is:

Regular-Expressions.info (quick and detailed reference info available)
https://www.regular-expressions.info/refquick.html

Regular Expressions are a complex subject. Please take care when you construct rewriting rules, and test them thoroughly using the test facility included in the Configurator.

Some examples of Regular Expression usage are available in the following articles:

Q10813: What are some examples of creating header re-write rules?
Q10415: How do I notify senders of my new company e-mail address standard?

Search the Knowledge Base for "Header Rewrite" to find additional examples.

Regular Expression Syntax

Note to advanced users: These products support a limited set of Regular Expression features.

Lookbehind is available in SEG 7.2 (SPE 3.6) and above. It is not available in earlier versions.

This Regular Expression Parser (Regex++), and its documentation, are Copyright ©1998-2000 Dr John Maddock. Permission to use, copy, modify, distribute and sell Regex++ and its documentation for any purpose is hereby granted without fee, provided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in supporting documentation. Dr John Maddock makes no representations about the suitability of this software for any purpose. It is provided "as is" without express or implied warranty.

Literals
All characters are literals except: . * ? + ( ) { } [ ] ^ < and $ These characters are literals when preceded by a \ character. A literal is a character that matches itself (it does not have any control significance). (Note: < is used for lookbehind and was added to the list of control characters at SEG 7.2.)

Wildcard
The dot character . matches any single character.

Repeats
A repeat is an expression that is repeated an arbitrary number of times. An expression followed by "*" can occur any number of times including zero. An expression followed by "+" can occur any number of times, but at least once. An expression followed by "?" may occur zero or one times only. When it is necessary to specify the minimum and maximum number of repeats explicitly, the bounds operator "{}" may be used, thus "a{2}" is the letter "a" repeated exactly twice, "a{2,4}" represents the letter "a" repeated between 2 and 4 times, and "a{2,}" represents the letter "a" repeated at least twice with no upper limit. Note that there must be no white-space inside the {}, and there is no upper limit on the values of the lower and upper bounds. All repeat expressions refer to the shortest possible previous sub-expression: a single character; a character set, or a sub-expression grouped with "()" for example.

"ba*" will match all of "b", "ba", "baaa" etc.
"ba+" will match "ba" or "baaaa" for example but not "b".
"ba?" will match "b" or "ba".
"ba{2,4}" will match "baa", "baaa" and "baaaa".

Non-greedy repeats
Non-greedy repeats are possible by appending a '?' after the repeat; a non-greedy repeat is one which will match the shortest possible string.

For example to match html tag pairs one could use something like:

"\<\s*tagname[^>]*>(.*?)\<\s*/tagname\s*>"

In this case $1 will contain the text between the tag pairs, and will be the shortest possible matching string.

Parenthesis
Parentheses serve two purposes, to group items together into a sub-expression, and to mark what generated the match. For example the expression "(ab)*" would match all of the string "ababab". It is permissible for sub-expressions to match null strings. Sub-expressions are indexed from left to right starting from 1, sub-expression 0 is the whole expression.

Non-Marking Parenthesis
Sometimes you need to group sub-expressions with parentheses, but don't want the parentheses to spit out another marked sub-expression. In this case a non-marking parenthesis (?:expression) can be used. For example the following expression creates no sub-expressions:

"(?:abc)*"

Alternatives
Alternatives occur when the expression can match either one sub-expression or another. Each alternative is separated by a "|". Each alternative is the largest possible previous sub-expression; this is the opposite behavior from repetition operators.

"a(b|c)" could match "ab" or "ac".
"abc|def" could match "abc" or "def".

Sets
A set is a set of characters that can match any single character that is a member of the set. Sets are delimited by "[" and "]" and can contain literals, character ranges, character classes, collating elements and equivalence classes. Set declarations that start with "^" contain the compliment of the elements that follow.

Character literals
"[abc]" will match either of "a", "b", or "c".
"[^abc] will match any character other than "a", "b", or "c".

Character ranges
"[a-z]" will match any character in the range "a" to "z".
"[^A-Z]" will match any character other than those in the range "A" to "Z".

Note: Character ranges are highly locale dependent: they match any character that collates between the endpoints of the range.

Character classes are denoted using the syntax "[:classname:]" within a set declaration, for example "[[:space:]]" is the set of all whitespace characters. . The available character classes are:

alnum	Any alpha numeric character.
alpha	Any alphabetical character a-z and A-Z. Other characters may also be included depending upon the locale.
blank	Any blank character, either a space or a tab.
cntrl	Any control character.
digit	Any digit 0-9.
graph	Any graphical character.
lower	Any lower case character a-z. Other characters may also be included depending upon the locale.
print	Any printable character.
punct	Any punctuation character.
space	Any whitespace character.
upper	Any upper case character A-Z. Other characters may also be included depending upon the locale.
xdigit	Any hexadecimal digit character, 0-9, a-f and A-F.
word	Any word character - all alphanumeric characters plus the underscore.
unicode	Any character whose code is greater than 255, this applies to the wide character traits classes only.

There are some shortcuts that can be used in place of the character classes:

\w in place of [:word:]
\s in place of [:space:]
\d in place of [:digit:]
\l in place of [:lower:]
\u in place of [:upper:]

To include a literal "-" in a set declaration then: make it the first character after the opening "[" or "[^", the endpoint of a range, a collating element, or precede with an escape character as in "[\-]". To include a literal "[" or "]" or "^" in a set then make them the endpoint of a range, a collating element, or precede with an escape character.

Line anchors
An anchor matches the null string at the start or end of a line: "^" matches the null string at the start of a line, "$" matches the null string at the end of a line. Use these anchors to search for text at the start or end of a line.

Note: SMTP header lines can include spaces at the beginning of the line. The ^ does not match these spaces. See article Q16535.

Back references
A back reference is a reference to a previous sub-expression that has already been matched, the reference is to what the sub-expression matched, not to the expression itself. A back reference consists of the escape character "\" followed by a digit "1" to "9", "\1" refers to the first sub-expression, "\2" to the second etc.

For example the expression "(.*)\1" matches any string that is repeated about its mid-point for example "abcabc" or "xyzxyz". A back reference to a sub-expression that did not participate in any match, matches the null string: NB this is different to some other regular expression matchers.

Lookahead and Lookbehind
These features allow you to search for a string that is preceded or followed by another string (or NOT preceded or followed by the string, known as "negative" lookahead or lookbehind). The "ahead" or "behind" string is not part of the expression and is not consumed (the location of evaluation in the string does not move).

Lookahead uses the operator (?=x) (where x is the lookahead text) or (?!=x) for negative lookahead.
Lookbehind uses the operator (?<=x) (where x is the lookbehind text) or (?<!x) for negative lookbehind.

An example of a positive lookahead expression is b(?=a) (matches the b in bad but not the b in bed)

An example of a negative lookbehind expression is (?<!a)b (matches the b in bed but not the b in cab)

Note: Lookbehind is available in SEG 7.2 (SPE 3.6) but not in earlier versions. Be sure to test with the product version you are using.

Characters by code
This is an extension to the algorithm that is not available in other libraries. It consists of the escape character followed by the digit "0" followed by the octal character code. For example "\023" represents the character whose octal code is 23. Where ambiguity could occur use parentheses to break the expression up: "\0103" represents the character whose code is 103, "(\010)3 represents the character 10 followed by "3". To match characters by their hexadecimal code, use \x followed by a string of hexadecimal digits, optionally enclosed inside {}, for example \xf0.

Word operators
The following operators are provided for compatibility with the GNU regular expression library.

"\w" matches any single character that is a member of the "word" character class, this is identical to the expression "[[:word:]]".
"\W" matches any single character that is not a member of the "word" character class, this is identical to the expression "[^[:word:]]".
"\<" matches the null string at the start of a word.
"\>" matches the null string at the end of the word.
"\b" matches the null string at either the start or the end of a word.
"\B" matches a null string within a word.

The start of the sequence passed to the matching algorithms is considered to be a potential start of a word. The end of the sequence passed to the matching algorithms is considered to be a potential end of a word.

Buffer operators
The following operators are provide for compatibility with the GNU regular expression library, and Perl regular expressions:

"\`" matches the start of a buffer.
"\A" matches the start of the buffer.
"\'" matches the end of a buffer.
"\z" matches the end of a buffer.
"\Z" matches the end of a buffer, or possibly one or more new line characters followed by the end of the buffer.

A buffer is considered to consist of the whole sequence passed to the matching algorithms.

Escape operator
The escape character "\" has several meanings.

Inside a set declaration the escape character is a normal character unless the flag regbase::escape_in_lists is set in which case whatever follows the escape is a literal character regardless of its normal meaning.

The escape operator may introduce an operator for example: back references, or a word operator.

The escape operator may make the following character normal, for example "\*" represents a literal "*" rather than the repeat operator.

Single character escape sequences

Escape sequence	Character code	Meaning
\e	0x1B	ASCII Escape character.
\xXX	0xXX	A hexadecimal character code, where XX is one or more hexadecimal digits.
\0dd	0dd	An octal character code, where dd is one or more octal digits.

Miscellaneous escape sequences

The following are provided mostly for Perl compatibility, but note that there are some differences in the meanings of \l \L \u and \U:

\w	Equivalent to [[:word:]].
\W	Equivalent to [^[:word:]].
\s	Equivalent to [[:space:]].
\S	Equivalent to [^[:space:]].
\d	Equivalent to [[:digit:]].
\D	Equivalent to [^[:digit:]].
\l	Equivalent to [[:lower:]].
\L	Equivalent to [^[:lower:]].
\u	Equivalent to [[:upper:]].
\U	Equivalent to [^[:upper:]].
\C	Any single character, equivalent to '.'.
\X	Match any Unicode combining character sequence, for example "a\x 0301" (a letter a with an acute).
\Q	The begin quote operator, everything that follows is treated as a literal character until a \E end quote operator is found.
\E	The end quote operator, terminates a sequence begun with \Q.

What gets matched
The regular expression library will match the first possible matching string. If more than one string starting at a given location can match then it matches the longest possible string. In cases where their are multiple possible matches all starting at the same location, and all of the same length, then the match chosen is the one with the longest first sub-expression. If that is the same for two or more matches, then the second sub-expression will be examined and so on.

Substitution Syntax
When generating replacement fields the following substitution syntax is supported.

In format strings, all characters are treated as literals except: ()$\?:
To use any of these as literals you must prefix them with the escape character \

The following special sequences are recognized:

Grouping

Sub-expression expansion

$`	Expands to all the text from the end of the previous match to the start of the current match, if there was no previous match in the current operation, then everything from the start of the input string to the start of the match.
$'	Expands to all the text from the end of the match to the end of the input string.
$&	Expands to all of the current match.
$0	Expands to all of the current match.
$N	Expands to the text that matched sub-expression N.