This article applies to:
- WebMarshal 6.X
- WebMarshal 7.X
- Security Reporting Center 2.X
- WebTrends Firewall Suite
Question:What is the WELF log file format?
WELF is the WebTrends Enhanced Log file Format.
The WELF Reference defines the WebTrends industry standard log file exchange format. Any firewall or VPN system logging to this format will be compatible with Firewall Suite 2.0 and later, Firewall Reporting Center 1.0 and later, and Security Reporting Center 2.0 and later.
WebMarshal 6.X and WebMarshal 7.X "Traffic Logging" logs can be created in the WELF format.
- For full details of the fields logged by WebMarshal, see Trustwave Knowledgebase article Q21119.
Log File Format
A log file is made up of records. Each record makes up a single line of the file. Records must be in chronological order. The earliest record is the first record in the file; the most recent record is the last record in the file. The WebTrends Enhanced Log Format places no restrictions on log file names or log file rotation policies.
A record is terminated by the character sequence carriage return-line feed (0x0D-0x0A). There may be no carriage-returns or line-feeds within a record; this format results in a single record per line.
Each record is made up of fields. The record identifier field (id=) must be the first field in a record. All other fields can appear in any order.
Aside from a few required fields, you can decide which optional fields are included in the record. You may want some fields to appear in only certain records because they are only relevant to certain types of activity (for example, the operation on an HTTP request).
Some optional fields may be left out if the firewall vendor chooses, but doing so typically results in reports that are less complete. Refer to the field descriptions to determine which fields are required for tables.
(In a real log file, the record would reside on one line.)
Note: This sample record should give you a sense of what a record looks like. It does not contain all the fields that are available and described in this document. Additional sample records are fully documented in the HELP Index built into the Firewall Suite product.
id=firewall time="2000-2-4 12:01:01" fw=192.168.0.238 pri=6 rule=3 proto=http src=192.168.0.23 dst 18.104.22.168 rg=www.webtrends.com/index.html op=GET result 0 rcvd=1426
- Record identifier
The id= field identifies the type of record. For log files conforming to this document, the type will always be firewall. For example,
yyyy: year (always 4 digits)
mm: number between 1 and 12 (inclusive) to represent the month (1 or 2 digits)
dd: day of the month, 1 based (1 or 2 digits)
hh: hour, based on 24-hour clock (1 or 2 digits)
mm: minute (1 or 2 digits)
ss: second (1 or 2 digits)
6:00 a.m. on January 1, 2000 would be represented as:
It could also be represented as:
6:00 p.m. on the same day would be represented as:
- Priority of the record
The pri= field specifies the priority of the event. The following is a list of valid values:
0 - emergency
1 - alert
2 - critical
3 - error
4 - warning
5 - notice
6 - information
7 - debug
Messages are placed in various tables based on the priority. Messages with priorities 0, 1, and 2 are included in the critical errors tables, messages with priorities of 3 and 4 are included in the errors and warnings tables, and messages with priorities of 5, 6, and 7 are included in the informational messages tables. For example:
The following fields for the WebTrends Enhanced Log File Format are optional:
The rule= field specifies the rule that triggered the log entry. This field is used to generate tables that help the user understand that the rules they have set up are working properly. Three tables are based on the rule field: internal IP addresses triggering firewall rules, external IP addresses triggering firewall rules, and protocols triggering firewall rules. Most firewalls log this field as an integer identifying a particular rule. However, rules could also be identified by name and logged as such in this field. For example:
The following is an extract from the wtprotocols.txt file that ships with WebTrends Firewall Suite:
Map new protocols
Your log files may contain protocols not included in this file. Map them using the Protocols tab in the Firewall Options dialog in the GUI (a mapping changes file named protocols.txt is created). Or you can create the protocols.txt file and map new protocols. Follow the syntax of the wtprotocols.txt file: use the types of traffic designations enclosed in square brackets and list new protocols for that type of traffic, each on a single line. Note: Unmapped protocols are grouped for reports in a type of traffic designation called "other."
The duration= field specifies the time that is required to perform the operation, in seconds. For example, for an FTP file transfer, the duration is the amount of time used to perform the transfer. Although Firewall Suite tracks this field, it is not currently shown in any tables or graphs. We recommend that if the this information is available, it should be logged so that it can be used in the future. For example, to indicate that an operation required 3 minutes exactly, the duration field could look like this:
or like this:
The sent= field specifies the number of bytes transferred from the source to the destination. For example,
The rcvd= field specifies the number of bytes transferred from the destination to the source. For example:
The src= field specifies the IP address that generated the event. For example:
The srcname= fields is a more user-friendly version of the src= field. For example,
The dst= field specifies the IP address that received the event. For example,
The dstname= field is a more user-friendly version of the dst= field. For example,
If users are authenticating through the firewall, then the authenticated user name can be logged in the user= field. For example,
For HTTP and FTP requests, the op= field is the operation such as GET, POST, etc. For example,
For HTTP and FTP requests, the arg= field is the URL accessed. For example,
For HTTP requests, the result= field is the standard result code, such as 200 for success, 304 for returned from cache, etc. For example,
The vpn= field identifies a particular VPN. This value is used to generate tables showing the most highly used VPNs and tables correlating particular users to particular VPNs. For example,
vpn="NY Branch VPN"
The firewall vendor can use the type= field to cause records to be placed into the tables relating to VPN events or relating to firewall management events. (Other categories may be defined in the future.)
A record can be put into more than one category by separating values by commas.
The currently defined types are:
· vpn - the record is a VPN event.
· mgmt - the record is a firewall management event. For example,
The msg= field is the basis for the tables showing detailed Critical Events, Errors and Warnings, VPN events, and Firewall Management events. Firewall Suite generates summary tables showing these types of events. Firewall Suite will also generate detailed tables associating users with these events. To make this happen, the user(s) need to be identified using the Src=, Srcname=, Dst=, Dstname=, or User= fields. For example,
msg="Possible port scan detected"
msg="Firewall configuration changed"
For incoming web records, the ref= field contains the referring site. For example,
For incoming or outgoing web records, the agent= field contains the agent (usually the browser).
agent="SPRY_Mosaic/v8.32 (Windows 16-bit)"
agent="Microsoft Internet Explorer/4.40.308 (Windows 95)"
agent="Mozilla/3.0 (WinNT; I)"
The cat= field contains the categories to which the accessed site belongs. It is used only for firewalls or proxies capable of categorizing web sites. For example, www.msnbc.com might be categorized as "General News", "Investment", and "Entertainment". If a site belongs to more than one category, these categories should be given in the same cat= field, with a comma separating each category. Note: If a field contains spaces, it must be enclosed in double quotes.
Use this field should only if dst= (or dstname=) is also present.
dst=www.msnbc.com cat="General News"
The cat_action= field contains the action taken for the category value of the cat= field. For example, access to gambling web sites may be blocked.
This field should only be present if cat= is also present.
Possible values for this field are: block and pass.
dst=www.gambling.com cat=Gambling cat_action=block
dst=www.msnbc.com cat=News cat_action=pass
The wmcache= field contains the result of a WebMarshal cache lookup request.
- Cache status is not written for HTTPS or FTP requests. Only HTTP supports caching.
- This custom field cannot be reported by Security Reporting Center.
Possible values for this field are: HIT, MISS, REFRESH_HIT or REFRESH_MISS.
- wmcache=HIT indicates that the item was served from cache without checking the origin server.
- wmcache=MISS indicates that the item was not in cache and had to be retrieved from the origin server.
- wmcache=REFRESH_HIT indicates that the cache item required revalidation, and that revalidation was successful.
- wmcache=REFRESH_MISS indicates that the cache item required revalidation, and that the origin server sent back new data.
Two fields record the result of WebMarshal classification:
dclass= indicates a WebMarshal Domain Classification
fclass= indicates a WebMarshal File Classification
If WebMarshal records multiple classifications, they are included as a comma separated list within double quotes.
For instance: dclass="Safe Sites,Search Engines"
See the document entitled WebTrends Enhanced Log Format (WELF) For Firewalls & VPNs or the HELP Index built into the Firewall Suite product for examples of records that conform to the WebTrends Enhanced Log File Format. Included among the examples provided are:
- Sample Web Records
- Sample E-mail Records
- Sample Telnet Records
- Sample FTP Records
- Sample RealAudio Records
- Sample Management Records
- Sample Error Messages
- This article was previously published as: