Ubuntuland

Media Configuration Guidelines

Dedicated or shared backup environment
One design decision is whether to make your backup environment dedicated or shared.
Dedicated SANs are secure but expensive. Shared environments cost less, but require
more work to make them secure. A SAN installation with a database may require the
performance of a RAID 1 array. An installation backing up a basic file structure may
satisfy its needs with RAID 5 or NAS.

Pooling
Here are some useful conventions for media pools (formerly known as volume pools):
Configure a scratch pool for management of scratch tapes. If a scratch pool exists, EMM can move volumes from that pool to other pools that do not have volumes available.
Use the available_media script in the goodies directory. You can put the available_media report into a script which redirects the report output to a file and emails the file to the administrators daily or weekly. This helps track which tapes are full, frozen, suspended, and so on. By means of a script, you can also filter the output of the available_media report to generate custom reports.

To monitor media, you can also use the NetBackup Operations Manager (NOM). For instance, NOM can be configured to issue an alert if there are fewer than X number of media available, or if more than X% of the media is frozen or suspended.
Use the none pool for cleaning tapes.
Do not create too many pools. The existence of too many pools causes the library capacity to become fragmented across the pools. Consequently, the library becomes filled with many partially-full tapes.

Disk versus tape
Disk is becoming more common as a backup medium. Storing backup data on disk
generally provides faster restore.
Tuning disk-based storage for performance is similar to tuning tape-based storage. The optimal buffer settings for a site can vary according to its configuration. It takes thorough testing to determine these settings.

Disk-based backup storage can be useful if you have a lot of incremental backups and the percentage of data change is small. If the volume of data in incremental copies is insufficient to ensure streaming to tape drives, writing to disk can speed the backup process and alleviate wear and tear on your tape drives.
Here are some factors to consider when choosing to back up a given dataset to disk or tape:
Short or long retention period
Incremental or full backup
Intermediate (staging) or long-term storage
Delay in recovery time
Here are some benefits of backing up to disk rather than tape:
No need to multiplex
Writing to disk does not need to be streamed. This means that multiplexing is not necessary.
Multiplexing is only necessary with tape because the tape must be streamed.
Multiplexing allows multiple clients and multiple file systems to be backed up to the same tape simultaneously, thus streaming the drive. However, this functionality slows down the restore.

Instant access to data
Most tape drives on the market have a “time to data” of close to two minutes. This time includes the amount of time to move the tape from its slot, load it into the drive and seek an appropriate place on tape. Disk has an effective time to da ta of zero seconds. To understand the significance of eliminating this delay, consider that restoring a large file system whose backups reside on 30 different tapes means that a two-minute delay per tape adds almost two hours to the restore.
This includes the time it takes to eject and unload the 30 tapes.

Fewer full backups.
With tape-based systems, full backups must be done regularly because of the instant access to data issue described above. Otherwise, the number of tapes required for a restore significantly increases both the time to restore and the chance that a single tape will cause the restore to fail. Since disk arrays are protected by RAID software, they do not have this problem.

Introduction
Before you create a database, decide how to protect the database against potential failures.
Answer the following questions before developing your backup strategy.
Is it acceptable to lose any data if a hardware failure damages some of the files that constitute a database?
Will you ever need to recover to past points-in-time?
Does the database need to be available at all times (24x7)?
For specific information on backing up and restoring your database, refer to the NetBackup administrator ’s guide for your database product. In addition, the manufacturer of your database product may provide publications that document backup recommendations and methods.

Considerations for database backups
When planning your database backups, consider the following.

Fragmentation and databases
Using a smaller fragment size in a backup of a database such as Oracle will not improve backup performance, and may hinder restore performance. Database backups (when not using Advanced Client) are unaffected by fragmentation since there is only one “file” per backup image. There is no advantage in tape positioning with or without fast-locate blocks.

Using Advanced Client
NetBackup Advanced Client provides snapshot backup technology combined with off-host data movement for local networks and SAN environments. A data snapshot can be created on disk in seconds and then backed up directly to tape. Users can significantly reduce CPU and I/O overhead from application or database servers while eliminating the backup window altogether.
Advanced Client helps reduce the impact on applications that require 24x7xforever availability. Advanced Client is a vailable on UNIX and Windows systems, and supports all NetBackup libraries and drives. It can be used with multi-streaming and multiplexing, and with a variety of disk arrays.

Best practices: new tape drive technologies
VERITAS provides a white paper on best practices for migrating your NetBackup installation to new tape technologies:
“Best Practices: Migrating to or Integrating New Tape Drive Technologies in Existing Libraries,” available at www.support.veritas.com.
Recent tape drives offer noticeably higher capacity than the previous generation of tape drives targeted at the open-systems market. Administrators may want to take advantage of these higher-capacity, higher performance tape drives, but are concerned about integrating these into an existing tape library. The white paper discusses different methods for doing so and the pros and cons of each.

Best practices: tape drive cleaning
This section discusses several ways to clean tape drives. Refer to the NetBackup Media Manager System Administrator ’s Guide for details on how to use the methods discussed here.
Note The TapeAlert feature is discussed in detail later in this section.
Here are the tape drive cleaning methods that can be used in a NetBackup installation:
Frequency-based cleaning
On-demand cleaning
Tape A lert
Robotic cleaning

Frequency-based cleaning
NetBackup does frequency-based cleaning by tracking the number of hours a drive has been in use. When this time reaches a configurable parameter, NetBackup creates a job that mounts and exercises a cleaning tape. This cleans the drive in a preventive fashion.
The advantage of this method is that typically there are no drives unavailable awaiting cleaning. There is also no limitation on platform or robot type. On the downside, cleaning is done more often than necessary. This adds system wear and consumes time that could be used to write to the drive. Another limitation is that this method is hard to tune. When new tapes are used, drive cleaning is needed less frequently; the need for cleaning increases as the tape inventory ages. This increases the amount of tuning administration needed and, consequently, the margin of error.

On-demand cleaning
Refer to the NetBackup Media Manager System Administrator’s Guide for more information
on this topic.
Ta p e Al e r t
TapeAlert allows reactive cleaning for most drive types. TapeAlert allows a tape drive to notify EMM when it needs to be cleaned. EMM then performs the cleaning. You must have a cleaning tape configured in at least one library slot in order to utilize this feature.
TapeAlert is the recommended cleaning solution if it can be implemented.
Not all drives, at all firmware levels, support this type of reactive cleaning. In the case where reactive cleaning is not supported on a particular drive, frequency-based cleaning may be substituted. This solution is not vendor or platform specific. The specific firmware levels have not been tested by VERITAS, however the vendor should be able to confirm that the TapeAlert feature is supported.
How TapeAlert works
To understand NetBackup's behavior with drive-cleaning TapeAlerts, it is important to understand the TapeAlert interface to a drive. The TapeAlert interface to a tape drive is via the SCSI bus, based on a Log Sense page, which contains 64 alert flags. The conditions that cause a flag to be set and cleared are device-specific and are determined by the device vendor.
The configuration of the Log Sense page is via a Mode Select page. The Mode Sense/Select configuration of the TapeAlert interface is compatible with the SMART diagnostic standard for disk drives.
NetBackup reads the TapeAlert Log Sense page at the beginning and end of a write/read job. TapeAlert flags 20 to 25 are used for cleaning management although some drive vendors’ implementations may vary from this. NetBackup uses TapeAlert flag 20 (Clean Now) and TapeAlert flag 21 (Clean Periodic) to determine when it needs to clean a drive.
When a drive is selected by NetBackup for a backup, the Log Sense page is reviewed by bptm for status. If one of the clean flags is set, the drive will be cleaned before the job starts.
If a backup is in progress and one of the clean flags is set, the flag is not read until a tape is dismounted from the drive.
If a job spans media and, during the first tape, one of the clean flags is set, the cleaning light comes on and the drive will be cleaned before the second piece of media is mounted in the drive.

Best practices: storing tape cartridges
The implication is that the present job will conclude its ongoing write despite a TapeAlert Clean Now or Clean Periodic message. That is, the TapeAlert will not require the loss of what has been written to tape so far. This is true regardless of the number of NetBackup jobs involved in writing out the rest of the media.
Note that the behavior described here may change in the future.
If a large number of media become FROZEN as a result of having implemented TapeAlert, there is a strong likelihood of underlying media and/or tape drive issues.
Disabling TapeAlert
To disable TapeAlert, create a touch file called NO_TAPEALERT:
UNIX:
/usr/openv/volmgr/database/NO_TAPEALERT
Windo ws:
install_path\volmgr\database\NO_TAPEALERT

Robotic cleaning
Robotic cleaning is not proactive, and is not subject to the limitations detailed above. By being reactive, unnecessary cleanings are eliminated, frequency tuning is not an issue, and
the drive can spend more time moving data, rather than in maintenance operations.
Library-based cleaning is not supported by EMM for most robots, since robotic library and operating systems vendors have implemented this type of cleaning in many different ways.

Best practices: storing tape cartridges
A couple of issues with tape management are how long and where to store the tapes that you need to keep on site. Typically, a DLT tape can be stored for up to three years. The storage location should be climate-controlled and away from sunlight. In addition, the tapes should always be stored in their plastic boxes.

Best practices: recoverability
Recovering from data loss involves both planning and technology to support your recovery objectives and time frames. The following table, Methods and procedures for recoverability, describes how you can use NetBackup and other tools to recover from various mishaps or disaster. The methods and procedures you adopt for your installation should be documented and tested regularly to ensure that your installation can recover
from disaster.
Methods and procedures for recoverability
Operational Risk Recovery Possible? Methods and Procedures
File deleted before backup No None
File deleted after backup Yes Standard NetBackup restore procedures
Backup client failure Yes Data recovery using NetBackup
Media failure Yes Backup image duplication
Master/media server failure
Yes Manual failover to alternate server
Loss of backup database Yes NetBackup database recovery
No NetBackup software Yes If multiplexing was not used, recovery of media without NetBackup, using GNU tar
Complete site disaster Yes Vaulting and off site media storage

Suggestions for data recovery planning
It is important to have a well-documented and tested plan to recover from a logical error, an operator error, or a site disaster. The following practices have been found effective for recoverability in production environments. Refer also to the NetBackup Troubleshooting Guide and the NetBackup System Administrator's Guide for further information on disaster recovery.
Always use a regularly scheduled hot catalog backup
Refer to “Catalog Recovery from an Online Backup” in the NetBackup Troubleshooting Guide.
Review the disaster recovery plan often
Review your site-specific recovery procedures and verify that they are accurate and up-to-da te. Also, verify that the more complex systems, such as the NetBackup master and media servers, have current procedures for rebuilding the machines with the latest software.
Perform test recoveries on a regular basis
Implement a plan to perform restores of various systems to alternate locations. This plan should include selecting random production backups and restoring the data to a non-production system. A checksum can then be performed on one or many of the restored files and compared to the actual production data. Be sure to include offsite storage as part of this testing. The end-user or application administrator can also be involved in determining the integrity of the restored data.

Support NetBackup recoverability
Back up the NetBackup catalog to two tapes.
The catalog contains information vital for NetBackup recovery. Its loss could result in hours or days of recovery time through manual processes. The cost of a single tape is a small price to pay for the added insurance of rapid recovery in the event of an emergency.
Back up the catalog after each backup.
If a hot catalog backup is used, an incremental catalog backup can be done after each backup session. Extremely busy backup environments should also use a scheduled hot catalog backup, since their backup sessions end infrequently.
In the event of a catastrophic failure, the recovery of images is slowed by not having all images available. If a manual backup occurs just before the master server or the drive that contains the backed-up files crashes, the manual backup must be imported to recover the most recent version of the files.

Best practices: naming conventions
Record the IDs of catalog backup tapes.
Record the catalog tapes in the site run book or another public location to ensure rapid identification in the event of an emergency. If the catalog tapes are not identified ahead of time, a significant amount of time may be lost by scanning every tape in a library to find them.
The utility vmphyinv can be used to mount all tapes in a robotic library and identify the catalog tape(s). The vmphyinv utility will identify cold catalog tapes.
Designate label prefixes for catalog backups.
Make it easy to identify the NetBackup catalog data in times of emergency. Label the catalog tapes with a unique prefix such as “DB” on the tape barcod es, so your operators can find the catalog tapes without delay.
Place NetBackup catalogs in specific robot slots.
Place a catalog backup tape in the first or last slot of a robot to more easily identify the tape in an emergency. This also allows for easy tape movement if manual tape handling is necessary.
Put the NetBackup catalog on different online storage than the data being backed up.
In the case of a site storage disaster, the catalogs of the backed-up data should not reside on the same disks as production data. The reason behind this is straightforward: you want to avoid the case where, if a disk drive loses production data, it also loses the catalog of the production data, resulting in increased downtime.
Regularly confirm the integrity of the NetBackup catalog.
On a regular basis, such as quarterly or after major operations or personnel changes, walk through the process of recovering a catalog from tape. This essential part of NetBackup administration can save hours in the event of a catastrophe.

Use a consistent naming convention on all NetBackup master servers. Examples are provided below. Use lower case for all names. In most cases, the case will not cause issues, but some issues can occur when the installation comprises UNIX and Windows master and media servers.

Policy names
One good naming convention for policie is platform_datatype_server(s).
Example 1: w2k_filesystems_trundle
This policy name designates a policy for a single Windows server doing file system
backups.
Example 2: w2k_sql_servers
This policy name designates a policy for backing up a set of Windows 2000 SQL servers.
Several servers may be backed up by this policy. Servers that are candidates for being included in a single policy are those running the same operating system and with the same backup requirements. Grouping servers within a single policy reduces the number of policies and eases the management of NetBackup.

Schedule names
Create a generic scheme for schedule naming. One recommended set of schedule names is daily, weekly, and monthly. Another recommended set of names is incremental, cumulative, and full. This convention keeps the management of NetBackup at a
minimum. It also helps with the implementation of Vault, if your site uses Vault.
Storage unit/storage group names
A good naming convention for storage units is to name the storage unit after the media server and the type of data being backed up.
Two examples: mercury_filesystems and mercury_databases
where “mercury” is the name of the media server and “filesystems” and “databases”
identify the type of data being backed up.

Performance Tuning

Overview
The final measure of NetBackup performance is the length of time required for backup operations to complete (usually known as the backup window), or the length of time required for a critical restore operation to complete. However, to measure existing performance and improve future performance by means of those measurements calls for performance metrics more reliable and reproducible than simple wall clock time. This chapter will discuss these metrics in more detail.
After establishing accurate metrics as described here, you can measure the current performance of NetBackup and your system components to compile a baseline performance benchmark. With a baseline, you can apply changes in a controlled way. By measuring performance after each change, you can accurately measure the effect of each change on NetBackup performance.

Controlling system variables for consistent testing conditions
For reliable performance evaluation, eliminate as many unpredictable variables as possible in order to create a consistent backup environment. Only a consistent environment will produce reliable and reproducible performance measurements. Some of the variables to consider are described below as they relate to the NetBackup server, the network, the NetBackup client, or the data itself.
Server variables
It is important to eliminate all other NetBackup activity from your environment when you are measuring the performance of a particular NetBackup operation. One area to consider is the automatic scheduling of backup jobs by the NetBackup scheduler.
When policies are created, they are usually set up to allow the NetBackup scheduler to initiate the backups. The NetBackup scheduler will initiate backups based on the raditional NetBackup frequency-based scheduling or on certain days of the week, month, or other time interval. This process is called calendar-based scheduling. As part of the backup policy definition, the Start Window is used to indicate when the NetBackup scheduler can start backups using either frequency-based or calendar-based scheduling.
When you perform backups for the purpose of performance testing, this setup might interfere since the NetBackup scheduler may initiate backups unexpectedly, especially if the operations you intend to measure run for an extended period of time.

The simplest way to prevent the NetBackup scheduler from running backup jobs during your performance testing is to create a new policy specifically for use in performance
testing and to leave the Start Window field blank in the schedule definition for that policy.
This prevents the NetBackup scheduler from initiating any backups automatically for that policy. After creating the policy, you can run the backup on demand by using the Manual Backup command from the NetBackup Administration Console.
To prevent the NetBackup scheduler from running backup jobs unrelated to the
performance test, you may want to set all other backup policies to inactive by using the Deactivate command from the NetBackup Administration Console. Of course, you must reactivate the policies to start running backups again.
You can use a user-directed backup to run the performance test as well. However, using the Manual Backup option for a policy is preferred. With a manual backup, the policy contains the entire definition of the backup job, including the clients and files that are part of the performance test. Running the backup manually, straight from the policy, means there is no doubt which policy will be used for the backup, and makes it easier to change and test individual backup settings: from the policy dialog.
Before you start the performance test, check the Activity Monitor to make sure there is no NetBackup processing currently in progress. Similarly, check the Activity Monitor after the performance test for unexpected activity (such as an unanticipated restore job) that may have occurred during the test.
Additionally, check for non-NetBackup activity on the server during the performance test and try to reduce or eliminate it.

Note: By default, NetBackup logging is set to a minimum level. To gather more logging information, set the legacy and unified logging levels higher and create the appropriate legacy logging directories. For details on how to use NetBackup logging, refer to the logging chapter of the NetBackup Troublshooting Guide. Keep in mind that higher logging levels will consume more disk space.

Network variables
Network performance is key to achieving optimum performance with NetBackup. Ideally, you would use a completely separate network for performance testing to avoid the possibility of skewing the results by encountering unrelated network activity during the course of the test.
In many cases, a separate network is not available. Ensure that non-NetBackup activity is kept to an absolute minimum during the time you are evaluating performance. If possible, schedule testing for times when backups are not active. Even occasional short bursts of network activity may be enough to skew the results during portions of the performance test. If you are sharing the network with production backups occurring for other systems, you must account for this activity during the performance test.
Another network varia ble you must consider is host name resolution. NetBackup depends heavily upon a timely resolution of host names to operate correctly. If you have any delays in host name resolution, including reverse name lookup to identify a server name from an incoming connection from a certain IP address, you may want to eliminate that delay by using the HOSTS (Windows) or /etc/hosts (UNIX) file for host name resolution on systems involved in your performance test environment.

Client variables
Make sure the client system is in a relatively quiescent state during performance testing. A lot of activity, especially disk-intensive activity such as virus scanning on Windows, will limit the data transfer rate and skew the results of your tests.
One possible mistake is to allow another NetBackup server, such as a production backup server, to have access to the client during the course of the test. This may result in NetBackup attempting to back up the same client to two different servers at the same time, which would severely impact the results of a performance test in progress at that time.
Different file systems have different performance characteristics. For example, comparing data throughput results from operations on a UNIX VxFS or Windows FAT file system to those from operations on a UNIX NFS or Windows NTFS system may not be va lid, even if the systems are otherwise identical. If you do need to make such a comparison, factor the difference between the file systems into your performance evaluation testing, and into a ny conclusions you may draw from that testing.

Data variables
Monitoring the data you are backing up improves the repeatability of performance testing. If possible, move the data you will use for testing backups to its own drive or logical partition (not a mirrored drive), and defragment the drive before you begin performance testing. For testing restores, start with an empty disk drive or a recently defragmented disk drive with ample empty space. This will reduce the impact of disk fragmentation on the NetBackup performance test and yield more consistent results between tests.

Similarly, for testing backups to tape, always start each test run with an empty piece of media. You can do this by expiring existing images for that piece of media through the Catalog node of the NetBackup Administration Console, or by running the bpexpdate command. Another approach is to use the bpmedia command to freeze any media containing existing backup images so that NetBackup selects a new piece of media for the backup operation. This step will help reduce the impact of tape positioning on the NetBackup performance test and will yield more consistent results between tests. It will also reduce the impact of tape mounting and unmounting of media that has NetBackup catalog images and that cannot be used for normal backups.
When you test restores from tape, always restore from the same backup image on the tape to achieve consistent results between tests.
In general, using a large data set will generate a more reliable and reproducible performance test than a small data set. A performance test using a small data set would probably be skewed by startup and shutdown overhead within the NetBackup operation.
These variables are difficult to keep consistent between test runs and are therefore likely to produce inconsistent test results. Using a large data set will minimize the effect of start up and shutdown times.

Design the makeup of the dataset to represent the makeup of the data in the intended production environment. For example, if the data set in the production environment contains many small files on file servers, then the data set for the performance testing should also contain many small files. A representative test data set will more accurately predict the NetBackup performance that you can reasonably expect in a production environment.
The type of data can help reveal bottlenecks in the system. Files consisting of non-compressible (random) data cause the tape drive to run at its lower rated speed. As long as the other components of the data transfer path are keeping up, you may identify the tape drive as the bottleneck. On the other hand, files consisting of highly-compressible data can be processed at higher rates by the tape drive when hardware compression is enabled. This may result in a higher overall throughput and possibly expose the network as the bottleneck.
Many values in NetBackup provide data amounts in kilobytes and rates in kilobytes per second. For greater accuracy, divide by 1024 rather than rounding off to 1000 when you convert from kilobytes to megabytes or from kilobytes per second to megabytes per
second.

Evaluating performance
There are two primary locations from which to obtain NetBackup da ta throughput statistics: the NetBackup Activity Monitor and the NetBackup All Log Entries report. The choice of which location to use is determined by the type of NetBackup operation you are measuring: non-multiplexed backup, restore, or multiplexed backup.
You can obtain statistics for all three types of operations from the NetBackup All Log Entries report. You can obtain statistics for non-multiplexed backup or restore operations from the NetBackup Activity Monitor. For multiplexed backup operations, you can obtain the overall statistics from the All Log Entries report after all the individual backup operations which are part of the multiplexed backup are complete. In this case, the statistics available in the Activity Monitors for each of the individual backup operations are relative only to that operation, and do not reflect the actual total data throughput to the tape drive.
There may be small differences between the statistics ava ilable from these two locations
due to slight differences in rounding techniques between the entries in the Activity Monitor and the entries in the All Logs report. For a given type of operation, choose either the Activity Monitor or the All Log Entries report and consistently record your statistics only from that location. In both the Activity Monitor and the All Logs report, the data-streaming speed is reported in kilobytes per second. If a backup or restore is repeated, the reported speed can vary between repetitions depending on many factors, including the availability of system resources and system utilization, but the reported speed can be used to assess the performance of the data-streaming process.
The statistics from the NetBackup error logs show the actual amount of time spent reading and writing data to and from tape. This does not include time spent mounting and positioning the tape. Cross-referencing the information from the error logs with data from the bpbkar log on the NetBackup client (showing the end-to-end elapsed time of the entire process) indicates how much time was spent on operations unrelated to reading and writing to and from the tape.

To evaluate performance through the NetBackup Activity Monitor
1. Run the backup or restore job.
2. Open the NetBackup Activity Monitor.
3. Verify that the backup or restore job completed successfully.
The Status column should contain a zero (0).
4. View the log details for the job by selecting the Actions > Details menu option, or by double-clicking on the entry for the job. Select the Detailed Status tab.
5. Obtain the NetBackup performance statistics from the following fields in the Activity Monitor:
- Start Time/EndTime: These fields show the time window during which the backup or restore job took place.
- Elapsed Time: This field shows the total elapsed time from when the job was initiated to job completion and can be used as in indication of total wall clock time for the operation.
- KB per Second: This is the data throughput rate.
- Kilobytes: Compare this value to the amount of data. Although it should be comparable, the NetBackup data amount will be slightly higher because of administrative information, known as metadata, saved for the backed up data.
For example, if you display properties for a directory containing 500 files, each 1 megabyte in size, the directory shows a size of 500 megabytes, or 524,288,000 bytes, which is equal to 512,000 kilobytes. The NetBackup report may show 513,255 kilobytes written, reporting 1255 kilobytes more than the file size of the directory. This is true for a flat directory. Subdirectory structures may diverge due to the way the operating system tracks used and available space on the disk. Also, be aware that the operating system may be reporting how much space was
allocated for the files in question, not just how much data is actually there. For example, if the allocation block size is 1 kilobyte, 1000 1-byte files will report a total size of 1 megabyte, even though 1 kilobyte of data is all that exists. The greater the number of files, the larger this discrepancy may become.

To evaluate performance using the All Log Entries report
1. Run the backup or restore job.
2. Run the All Log Entries report from the NetBackup reports node in the NetBackup
Administrative Console. Be sure that the Date/Time Range that you select covers the time period during which the job was run.
3. Verify that the job completed successfully by searching for an entry such as “the
requested operation was successfully completed” for a backup, or “successfully read
(restore) backup id...” for a restore.
4. Obtain the NetBackup performance sta tistics from the following entries in the report.

Note The messages shown here will vary according to the locale setting of the master server.
Entry
Statistic
started backup job for client , policy , schedule
The Date and Time fields for this entry show the time at which the backup job started.
on storage unit successfully wrote backup id
For a multiplexed backup, this entry shows the size of the individual backup job and the
, copy , Kbytes
Date and Time fields show the time at which the job finished writing to the storage device.
The overall statistics for the multiplexed backup group, including the data throughput rate to the storage device, are found in a subsequent entry below.
successfully wrote of multiplexed backups, total Kbytes at Kbytes/sec
For multiplexed backups, this entry shows the overall statistics for the multiplexed backup
group including the data throughput rate.
successfully wrote backup id For non-multiplexed backups, this entry essentially combines the information in the previous two entries for multiplexed backups into one entry showing the size of the backup job, the data throughput rate, and the time, in the Date and Time fields, at which the job finished writing to the storage device.
, copy , fragment , Kbytes at Kbytes/sec the requested operation was successfully completed
The Date and Time fields for this entry show the time at which the backup job completed.
This value is later than the “successfully wrote” entry above because it includes extra processing time at the end of the job for tasks such as NetBackup image validation.

Entry
Statistic
begin reading backup id ,
The Date and Time fields for this entry show the time at which the restore job started reading from the storage device. (Note that the latter part of the entry is not shown for restores from disk, as it does not apply.)
(restore), copy , fragment from media id on drive index
successfully restored from backup id For a multiplexed restore (generally speaking, all restores from tape are multiplexed restores as non-multiplexed restores require additional action from the user), this entry shows the size of the individual restore job and the Date and , copy , Kbytes
Time fields show the time at which the job finished reading from the storage device. The overall statistics for the multiplexed restore group, including the data throughput rate, are found in a subsequent entry below.
successfully restored of For multiplexed restores, this entry shows the overall statistics for the multiplexed restore group, including the data throug hput rate.
requests , read total of Kbytes at Kbytes/sec successfully read (restore) backup id media , copy ,
For non-multiplexed restores (generally speaking, only restores from disk are treated as non-multiplexed restores), this entry essentially combines the information from the previous two entries for multiplexed restores into one entry showing the size of the restore job, the data throughput rate, and the time, in the Date and Time fields, at which the job finished reading from the storage device.
fragment , Kbytes at Kbytes/sec

Additional information
The NetBackup All Log Entries report will also have entries similar to those described above for other NetBackup operations such as image duplication operations used to create additional copies of a backup image. Those entries have a very similar format and may be useful for analyzing the performance of NetBackup for those operations.

The bptm debug log file for tape backups (or bpdm log file for disk backups) will contain the entries that are in the All Log Entries report, as well as additional detail about the operation that may be useful for performance analysis. One example of this additional detail is the intermediate da ta throughput rate message for multiplexed backups, as shown below:
... intermediate after successful, Kbytes at
Kbytes/sec
This message is generated whenever an individual backup job completes that is part of a multiplexed backup group. In the debug log file for a multiplexed backup group consisting of three individual backup jobs, for example, there could be two intermediate status lines, then the final (overall) throughput rate.
For a backup operation, the bpbkar debug log file will also contain additional detail about the operation that may be useful for performance analysis.
Keep in mind, however, that writing the debug log files during the NetBackup operation introduces some overhead that would not normally be present in a production environment. Factor that additional overhead into any calculations done on data captures while debug log files are in use.
The information in the All Logs report is also found in /usr/openv/netbackup/db/error (UNIX) or install_path\NetBackup\db\error (Windows).
See the NetBackup Troubleshooting Guide to learn how to set up NetBackup to write these debug log files.

Evaluating UNIX system components
In addition to evaluating NetBackup’s performance, you should also verify that common system resources are in adequate supply.
Monitoring CPU load
Use the vmstat utility to monitor memory use. Add up the “us” and “sy” CPU columns to get the total CPU load on the system (refer to the vmstat man page for details). The vmstat scan rate indicates the amount of swapping activity taking place.
The sar command also provides insight into UNIX memory usage.

Measuring performance independent of tape or disk output
It is possible to measure the disk (read) component of NetBackup’s speed independent of the network and tape components. There are two different techniques, described below.
The first, using bpbkar, is easier. The second may be helpful in more limited circumstances.
In these procedures, the master server is the client.
To measure disk I/O using bpbkar
1. Turn on the legacy bpbkar log by ensuring that the bpbkar directory exists.
UNIX: /usr/openv/netbackup/logs/bpbkar
Windo ws: install_path\NetBackup\logs\bpbkar
2. Set logging level to 1.
3. Enter the following:
UNIX
/usr/openv/netbackup/bin/bpbkar -nocont -dt 0 -nofileinfo
-nokeepalives filesystem > /dev/null
Where filesystem is the path being backed up.
Windo wsnstall_path\NetBackup\bin\bpbkar32 -nocont X:\ > NUL
Where X:\ is the path being backed up.
4. Check the time it took NetBackup to move the data from the client disk:
UNIX: The start time is the first PrintFile entry in the bpbkar log, the end time is the entry “Client completed sending data for backup,” and the amount of data is given in the entry Total Size.
Windows: Check the bpbkar log for the entry Elapsed time.
To measure disk I/O using the bpdm_dev_null touch file (UNIX only)
For UNIX systems, the procedure below can be useful as a follow-on to the bpbkar procedure (above). If the bpbkar procedure shows that the disk read performance is not the bottleneck and does not help isolate the problem, then the bpdm_dev_null procedure described below may be helpful. If the bpdm_dev_null procedure shows poor performance, the bottleneck is somewhere in the data transfer between the bpbkar process on the client and the bpdm process on the server. The problem may involve the network, or shared memory (such as not enough buffers, or buffers that are too small).

Caution If not used correctly, the following procedure can lead to data loss. Touching the bpdm_dev_null file red irects all disk backups to /dev/null, not just those backups using the storage unit created by this procedure. You should disable active production policies for the duration of this test and remove /dev/null as soon as this test is complete.
1. Enter the following:
touch /usr/openv/netbackup/bpdm_dev_null
Note The bpdm_dev_null file re-directs any backup that uses a disk storage unit to /dev/null.
2. Create a new disk storage unit, using /tmp or some other directory as the image directory path.
3. Create a policy that uses the new disk storage unit.
4. Run a backup using this policy. NetBackup will create a file in the storage unit directory as if this were a real backup to disk. This degenerate image file will be zero bytes long.
5. To remove the zero-length file and clear the NetBackup catalog of a backup that cannot be restored, run this command:
/usr/openv/netbackup/bin/admincmd/bpexpdate -backupid backupid -d 0
where backupid is the name of the file residing in the storage unit directory.
Evaluating Windows system components
In addition to evaluating NetBackup’s performance, you should also verify that common
system resources are in adequate supply. You may want to use the Windows Performance
Monitor utility included with Windows. For information about using the Performance Monitor, refer to your Microsoft documentation.
The Performance Monitor organizes information by object, counter, and instance.

An object is a system resource category, such as a processor or physical disk. Properties of an object are counters. Counters for the Processor object include %Processor Time, which is the default counter, and Interrupts/sec. Duplicate counters are handled via instances.
For example, to monitor the %Processor Time of a specific CPU on a multiple CPU
system, the Processor object is selected, then the %Processor Time counter for that object is selected, followed by the specific CPU instance for the counter.
When you use the Performance Monitor, you can view data in real time format or collect the data in a log for future analysis. Specific components to evaluate include CPU load, memory use, and disk load.

Monitoring CPU load
To determine if the system has enough power to accomplish the requested tasks, monitor the % Processor Time counter for the Processor object to determine how hard the CPU is working, and monitor the Process Queue Length counter for the System object to determine how many processes are actively waiting for the processor.
For % Processor Time, values of 0 to 80 percent are generally considered safe. Values from 80 percent to 90 percent indicate that the system is being pushed hard, while consistent values above 90 percent indicate that the CPU is a bottleneck.
Spikes approaching 100 percent are normal and do not necessarily indicate a bottleneck.
However, if sustained loads approaching 100 percent are observed, efforts to tune the system to decrease process load or an upgrade to a faster processor should be considered.
Sustained Processor Queue Lengths greater than two indicate too many threads are waiting to be executed. To correctly monitor the Processor Queue Length counter, the Performance Monitor must be tracking a thread-related counter. If you consistently see a queue length of 0, verify that a non-zero value can be displayed.

Note:The default scale for the Processor Queue Length may not be equal to 1. Be sure to read the data correctly. For example, if the default scale is 10x, then a reading of 40 actually means that only 4 processes are waiting.

Monitoring memory use
Memory is a critical resource for increasing the performance of backup operations. When you examine memory usage, view information on:
Committed Bytes. Committed Bytes displays the size of virtual memory that has been committed, as opposed to reserved. Committed memory must have disk storage available or must not require the disk storage because the main memory is large enough. If the number of Committed Bytes approaches or exceeds the amount of physical memory, you may encounter issues with page swapping.
Page Faults/sec. Page Faults/sec is a count of the page faults in the processor. A page fault occurs when a process refers to a virtual memory page that is not in its Working Set in main memory. A high Page Fault rate may indicate insufficient memory.

Monitoring disk load
To use disk performance counters to monitor the disk performance in Performance Monitor, you may need to enable those counters. Windows may not have enabled the disk performance counters by default for your system.
For more information about disk performance counters, from a command prompt, type:
diskperf -help
To enable these counters and allow disk monitoring:
Note On a Windows 2000 system, this is set by default.
1. From a command prompt, type:
diskperf -y
2. Reboot the system.
To disable these counters and cancel disk monitoring:
1. From a command prompt, type:
diskperf -n
2. Reboot the system.
When you monitor disk performance, use the %Disk Time counter for the PhysicalDisk object to track the percentage of elapsed time that the selected disk drive is busy servicing read or write requests.

Also monitor the Avg. Disk Queue Length counter and watch for values greater than 1 that last for more than one second. Values greater than 1 for more than a second indicate that multiple processes are waiting for the disk to service their requests.
Several techniques may be used to increase disk performance, including:
Check the fragmentation level of the data. A highly fragmented disk limits throughput levels. Use a disk maintenance utility to defragment the disk.
Consider adding additional disks to the system to increase performance. If multiple processes are attempting to log data simultaneously, dividing the data among multiple physical disks may help.
Determine if the data transfer involves a compressed disk. The use of Windows compression to automatically compress the data on the drive adds additional overhead to disk read or write operations, adversely affecting the performance of NetBackup. Use Windows compression only if it is needed to avoid a disk full condition.
Consider converting to a system based on a Redundant Array of Inexpensive Disks (RAID).
Though more expensive, RAID devices generally offer greater throughput, and, (depending on the RAID level employed), improved reliability.
Determine what type of controller technology is being used to drive the disk. Consider if a different system would yield better results. See the table “Drive controller data transfer rates” on page 12 for throughput rates for common controllers.