32.4 Configuring Access Gateway Logging

Logging HTTP transactions has associated costs. The Access Gateway is capable of handling thousands of transactions per second. If transaction volume is high and each log entry consumes a few hundred bytes, the Access Gateway can fill up the available disk space in a matter of minutes. HTTP logging also increases system overhead, which causes some degradation in performance. By default, the logging of HTTP transactions is turned off. Before enabling logging, you need to determine what needs to be logged and then plan a logging strategy.

32.4.1 Determining Logging Requirements

Because logging requirements and transaction volume vary widely, Novell cannot make recommendations regarding a specific logging strategy. The following tasks guide you through the process of creating a strategy that fits your business needs.

  1. Identify the reasons for tracking transactions such as customer billing, statistical analysis, or growth planning.

  2. Determine which resources need logging.

    You enable logging at the proxy service level. If you have a proxy service protecting resources whose transactions do not need to be logged, reconfigure your proxy services so that the proxy service you configure for logging contains only the resources for which you want to log transactions.

  3. Determine what information you need in each log entry.

    The common configuration for a log entry contains minimal information: the date, time, and client IP address for each entry. If you need more information, you can to select the extended log configuration. Do not select all available fields, but carefully select what you really need. For example, you can include cookie information, but cookie information can consume a large amount of space and might not include any critical information you need.

    You should log only the essential data because a few bytes can add up quickly when the Access Gateway is tracking thousands of hits every second. For information about what is available in an extended log profile, see Section 32.4.5, Configuring Extended Log Options.

  4. Design a rollover strategy.

    A log must be closed before it can be downloaded to another server for analysis or deleted. You specify either by time or size when the Access Gateway closes a log file and creates a new one. For each proxy service that you enable for logging, you need to reserve enough space for at least two files: one for logging and one for roll over. To calculate the best procedure, see Section 32.4.2, Calculating Rollover Requirements.

  5. Design a log deletion strategy

    The Access Gateway has a limited amount of disk space allocated for logging, and you need to decide how you are going to manage this space. You can limit the number of rollover files by number or age. You can also select to copy the files to another server and then delete them. To calculate the best procedure, see Section 32.4.2, Calculating Rollover Requirements.

32.4.2 Calculating Rollover Requirements

You can have the Access Gateway roll over log files based on time or on size, but not both. If you already know which option you want to use, scan this section and then complete only the calculations pertinent to your choice. If you don’t know which option best matches your situation, completing the calculations in this section should help you decide.

The following variables are used in the formulas:

  • logpartition_size: The total disk capacity reserved for log files on the Access Gateway.

    The Access Gateway reserves 4 GB to share between logging and system files. The system files do not grow significantly, so you can assume that you have about 2 GB for logging. To increase this size, see Section 32.4.7, Configuring the Size of the Log Partition.

  • logentry_size: The average log entry size.

    You can determine this by configuring a proxy service to track the required information, generating traffic to the proxy service, downloading the log files, determining how large each entry is, and calculating the average.

  • request_rate: The peak rate of requests per second.

    You can estimate this rate or place your Access Gateway in service and get more accurate data by accessing generated statistics. See Section 33.2, Monitoring Access Gateway Statistics.

  • num_services: The number of proxy services for which you plan to enable logging.

  • logs_per_service: The number of log files, both active and closed, that you want the Access Gateway to generate for each proxy service before the disk fills.

    You must plan to have at least two logs per proxy service, but you can have three or more.

The following formulas can help you estimate when the system would run out of resources:

Calculating diskfull_time

Using the following formula, you can calculate how long it will take the Access Gateway to fill your logging disk space:

diskfull_time in seconds = logpartition_size / (request_rate *
   logentry_size * num_services)

For example, assume the following:

  • logpartition_size = 1 GB (1,073,741,824 bytes)
  • request_rate = 1000 requests per second
  • logentry_size = 1 KB (1,024 bytes)
  • num_services = 1
diskfull_time = (1 GB) / (1000 * 1 KB * 1) = 1048 seconds (17.47
   minutes)

The logging disk space will fill up every 17.47 minutes.

To calculate the diskfull_time for your Access Gateway:

  1. Determine the values of the four variables listed above.

  2. Using the diskfull_time formula, calculate how often you can expect your logging disk to fill; then use the result in Calculating max_roll_time.

If your diskfull_time interval is too short to be practical for your rollover schedule, the easiest option is to reduce the log entry size by configuring the proxy services to log less information per transaction.

Calculating max_roll_time

Using the following formula, you can calculate the maximum rollover time value you should specify in the Roll over every field 

max_roll_time = diskfull_time / logs_per_service

For example, assume the following:

  • diskfull_time = 12 hours
  • logs_per_service = 2
max_roll_time = 12 / 2 = 6 hours

If you roll your logs over by time intervals, the maximum time should be less than six hours. Otherwise, scheduling the download and deletion of log files is much more complicated and the window in which this can be done is narrower.

To calculate the max_roll_time for your Access Gateway:

  1. Determine how many log files you want the Access Gateway to generate per service before log space fills.

    The minimum number is two.

  2. Using the max_roll_time formula and the diskfull_time value obtained in Calculating diskfull_time, calculate how often you should have the cache device roll over the log files.

  3. Record the max_roll_time result on your planning sheet.

Calculating max_log_roll_size

Using the following formula, you can calculate the maximum log file size you should specify in the Maximum File Size field: 

max_log_roll_size = logpartition_size / (num_services *
   logs_per_service)

For example, assume the following:

  • logpartition_size = 600 MB
  • num_services = 2
  • logs_per_service = 3
max_log_roll_size = 600 MB / (2 * 3) = 100 MB

If you roll your logs over when they reach a specific size, the file size must be no more than 100 MB. Otherwise, the system runs out of disk space before you have three complete log files and scheduling the download and deletion of log files is much more complex.

To calculate the max_log_roll_size for your Access Gateway:

  1. Determine the values of the three variables listed above.

  2. Using the max_log_roll_size formula, calculate the maximum size a log file should reach before the cache device rolls it over.

32.4.3 Enabling Logging

Do not enable logging until you have designed a logging strategy. See Section 32.4.1, Determining Logging Requirements.

  1. In the Administration Console, click Access Manager > Access Gateways > Edit > [Name of Reverse Proxy] > [Name of Proxy Service] > Logging.

    Logging configuration

  2. Fill in the following fields:

    Enable Logging: Select this field to enable logging.

    Stop Service On Log Failure: Select this field if you want the Access Gateway to deny requests to this proxy service because the Access Gateway cannot log entries for it.

    Log Directory: Displays the default location for the log files for this proxy service.

  3. In the Logging Profile List, click one of the following options:

    • New: Click this option to create a new logging profile. Then specify a name and select either Common or Extended.

    • Default: Click Default to modify or view the settings for the Default profile. The Default profile uses the common log options.

    A logging profile determines the type of information that is written to the log file; it also manages rollover and old file options.

  4. Continue with one of the following:

32.4.4 Configuring Common Log Options

Use the common log options page to control log rollover and old file options. The data included in a log entry is controlled by a default configuration that includes the following:

  • Date and time of the request

  • Username of the client

  • Remote host name

  • The request line as it came from the client

  • The HTTP status code returned to the client

  • The number of bytes in the document transferred to the client

The Access Gateway does not allow active log files to be deleted. Only log files that have been closed can be deleted. The rollover options allow you to control when a file is rolled over and closed, and a new file is created. The old file options allow you to control when the rolled-over log files are deleted.

To configure a default log file for a selected proxy service:

  1. Click Access Manager > Access Gateways > Edit > [Name of Reverse Proxy] > [Name of Proxy Service] > Logging > [Name of Common Log Profile].

    Configuring a common log profile

  2. Select one of the following roll over options:

    Maximum File Size: Rolls the file when it reaches the specified number of megabytes.

    Roll over every: Rolls the file at the specified interval. You can specify the interval in hours or days.

    • beginning: Specifies the day that the interval should begin. You can select a day of the week or the first of the month.

    • at: Select the hour of the day that the interval should begin and the time zone (either the local time zone or GMT).

  3. Select one of the following old file options:

    Maximum Number of Archived Files: Allows you to limit the number of old log files on the system to the number specified in this option. The oldest file is automatically deleted when this number is reached. All logging data in deleted files is lost. If you configure the Log Push option, you can set the system up so that the files are copied to another server before they are deleted from the server.

    Delete Files Older Than: Allows you to configure the Access Gateway to delete files when they are older than the time you specify. All logging data in deleted files is lost. If you configure the Log Push option, you can set the system up so that the files are copied to another server before they are deleted from the server.

    Do Not Delete: Prevents the system from automatically deleting the log files. You can use the Log Push option to copy the files to another server and then either manually delete them or have the Log Push option delete them from the server after they are copied to another server.

    For information about the Log Push option, see Section 32.4.6, Configuring Log Pushing.

  4. Click OK.

  5. Click the Access Gateways link, then click Update > OK.

32.4.5 Configuring Extended Log Options

Use the extended log options page to control log entry content, log rollover, and old file options. A log entry always includes the date, time, and client IP address for each entry, but with the log data options, you can add other fields such as the IP address of the server and the username of the client.

The Access Gateway does not allow active log files to be deleted. Only log files that have been closed can be deleted. The rollover options allow you to control when a file is rolled over and closed, and a new file is created. The old file options allow you to control when the rolled-over log files are deleted.

To configure an extended log file for a selected proxy service:

  1. Click Access Manager > Access Gateways > Edit > [Name of Reverse Proxy] > [Name of Proxy Service] > Logging > [Name of Extended Log Profile].

    Configuring an extended log profile

  2. Select one or more of the log data options:

    Name

    Description

    User Name

    The name of the user sending the request.

    Server IP

    The IP address of the Access Gateway.

    Site Name

    The name of the reverse proxy.

    Method

    The HTTP method the browser sent to the Access Gateway.

    URI

    The HTTP URL the browser sent to the Access Gateway.

    URI Stem

    The stem portion of the HTTP URL the browser sent to the Access Gateway. The stem is everything in the URL up to the first question mark. If the URL has no question mark, the URI Stem field is the same as the URI field. It is redundant if URI is selected.

    URI Query

    The query portion of the HTTP URL the browser sent to the Access Gateway. The query is everything from the first question mark through the end of the URL. If the URL has no question mark, this field has no value. It is redundant if URI is selected.

    Version

    The HTTP version specified in the URL the browser sent to the Access Gateway.

    Status

    The HTTP status code the Access Gateway sent to the browser.

    Bytes Sent

    The number of bytes of HTTP response data the Access Gateway sent to the browser.

    Bytes Received

    The number of bytes of HTTP request data the proxy service received from the browser.

    Time Taken

    The time in seconds it took the Access Gateway resources to deal with the request.

    User Agent

    The User-Agent HTTP request header value the browser sent to the Access Gateway.

    Cookie

    The Cookie HTTP request header value the browser sent to the Access Gateway. The Access Gateway doesn’t cache cookie information. Cookies can consume a lot of space. If you select this option, make sure it contains the critical information that you need.

    Referer

    The Referer HTTP request header value the browser sent to the Access Gateway.

    Cached Status

    The value indicates whether the request was filled from cache.

    • 1 = filled from cache
    • 0 = not filled from cache

    Fill Proxy

    The IP address of the upstream proxy.

    Origin Server

    The IP address of the Web server. This assumes the Access Gateway retrieved the requested information directly from the Web server.

    X-Forward-For

    The X-Forwarded-For HTTP request header value the browser sent to the Access Gateway. Do not confuse this with the X-Forwarded-For option that causes the Access Gateway to generate or forward headers to upstream proxies or Web servers.

    Bytes Filled

    (Linux only) The total bytes filled in response to the request.

    Fill Status

    (Linux only)

    Content Range

    (Linux only) The byte ranges sent from the Access Gateway to a requesting browser.

    E Tag

    (Linux only) The tag sent from the Access Gateway to a requesting browser.

    Completion Status

    (Linux only) The completion status for the transaction indicating that it completed successfully or that it failed. Possible values: success, timeout, reset (the client terminated the connection), administrative (the Access Gateway terminated the connection).

    Reply Header Size

    (Linux only) The size in bytes of the HTTP header associated with a response to a client.

    X Cache Info

    (Linux only) Brief status statement for cached objects; brief reasons why an object was not cached.

    Range

    (Linux only) The Range header value.

    If Range

    (Linux only) The If Range header value, which indicates whether the browser request was a conditional range request.

    Content Length

    (Linux only) The size in bytes of the entire object delivered to a requesting browser.

    Request Pragma

    (Linux only) The pragma value associated with a browser request.

    Reply Pragma

    (Linux only) The pragma value associated with a server response to a requesting browser.

  3. Select one of the following rollover options:

    Maximum File Size: Rolls the file when it reaches the specified number of megabytes.

    Roll over every: Rolls the file at the specified interval. You can specify the interval in hours or days.

    • beginning: Specifies the day that the interval should be begin. You can select a day of the week or the first of the month.

    • at: Select the hour of the day that the interval should begin and the time zone (either the local time zone or GMT).

  4. Select one of the following old file options.

    Maximum Number of Archived Files: Allows you to limit the number of old log files on the system to the number specified in this option. The oldest file is automatically deleted when this number is reached. All logging data in deleted files is lost. If you configure the Log Push option, you can set the system up so that the files are copied to another server before they are deleted from the server.

    Delete Files Older Than: Allows you to configure the Access Gateway to delete files when they are older than the time you specify. All logging data in deleted files is lost. If you configure the Log Push option, you can set the system up so that the files are copied to another server before they are deleted from the server.

    Do Not Delete: Prevents the system from automatically deleting the log files. You can use the Log Push option to copy the files to another server and then either delete them manually or have the Log Push option delete them from the server when they have been copied to another server.

    For information about the Log Push option, see Section 32.4.6, Configuring Log Pushing.

  5. Click OK.

  6. Click the Access Gateways link, then click Update > OK.

32.4.6 Configuring Log Pushing

(NetWare only) The Log Push option allows you to configure the NetWare Access Gateway to copy log files to an FTP server at specified intervals. The Log Push option is configured for all log files on the Access Gateway. If you have enabled logging on multiple proxy services, the Access Gateway uses the same configuration to push the log files of each proxy service.

This feature works within the following parameters:

  • The Access Gateway tries as many times as necessary to establish a connection with the FTP server during the hour of the scheduled push. When the hour changes, the Access Gateway stops trying until the next interval you have specified.

  • When the connection is established, the Access Gateway assumes that pushing the log files was successful. The Access Gateway does not detect any errors that prevent the successful pushing of the files.

For example, you specify that log files are to be pushed on every day of the week at 12 midnight. When the system clock reaches the target hour, the Access Gateway begins trying to establish a connection with the FTP server.

  • If a connection cannot be established before the hour changes to 1 a.m., the Access Gateway stops trying to connect and doesn't try again until 12 midnight the next day.

  • If a connection is established but an error occurs that prevents a successful push, the error is not detected, and the Access Gateway doesn't try to connect again until 12 midnight the next day.

To configure log pushing:

  1. In the Administration Console, click Access Manager > Access Gateways > Edit > Log Push.

    Configuring the Log Push option

  2. To enable log pushing, select Enable Log Push.

  3. Configure the following FTP settings. All of them are required settings.

    DNS or IP Address: Specify the DNS name or the IP address of your FTP server.

    Default Directory: Specify the directory on the FTP server to which the Access Gateway should copy the log files.

    Login Name: Specify the name that the Access Gateway should use to log in to the FTP server.

    Password: Specify the password that the Access Gateway should use for logging in.

  4. To schedule when the log files are copied to the FTP server, fill in the following fields:

    Cluster Member: (Available only if the Access Gateway is a member of a cluster.) Select the server you want to configure from the list of servers. The modifications made to the Push Using Address option apply only to the selected cluster member. Modifications made to any other options on the page apply to all members of the cluster.

    Push Using Address: Select the IP address you want to use for sending the log files to the FTP server.

  5. Select when you want the logs to be pushed. Select one of the following:

    • Push Logs when the Logs Roll Over: To push the logs as soon as a log file rolls over, select Push Logs when the Logs Roll Over.This method ensures that log files are copied as soon as possible.

    • Push Logs on Specified Days and Time: To push the logs on selected days at a specific time, select Push Logs on Specified Days and Time, then configure the following fields:

      Days to Push the Logs: Allows you to select the days when the log push should occur. You can select multiple days for pushing.

      Time to Push the Logs: Specifies the time of day when the log files are pushed.

  6. Specify what you want done with the log files after they have been copied to the FTP server.

    Select the Delete Log Files from Server after Push option to have the Access Gateway delete the log files after they have been copied to the FTP server. This is the recommended method. If you do not select this option, you must manually delete them or use the old file options on the Logging page (see Section 32.4.4, Configuring Common Log Options).

  7. Click OK.

  8. Click the Access Gateways link, then click Update > OK.

32.4.7 Configuring the Size of the Log Partition

The size of the log partition should be configured as part of the installation process. See one of the following in the Novell Access Manager 3.0 SP4 Installation Guide: