By Rich Luif, StorageTek and Michael Jones, SCSI Toolbox

Customer Service Engineers (CSEs) are called upon to perform a range of tasks on many products, in other words - be an expert “jack of all trades” in supporting the customer. In today's multi-platform integrated environment, it can be a difficult, time consuming, and error-prone exercise to determine the root cause of many problems.

Because computer storage underlies all systems today, it can be one of the first places to look if the system is not functioning properly. Is it an equipment problem? Bad media? Down revision firmware? If CSEs can not determine the cause quickly, the tendency is to send it back to the repair center. The ability to quickly and accurately test and diagnose storage devices in the field can save considerable time and a costly replacement.

That was the challenge faced by StorageTek's Global Services Business Group, StorageTek CSEs are talented and trained individuals, but they work on a wide range of products, and there is a lot of pressure to get the customer back on-line as quickly as possible. In the past, a storage device would usually be shipped back to the repair center where it would be diagnosed. But StorageTek was experiencing high NDF (No Defects Found) rates, and the associated cost of shipping and handling, disruption to the customer, and the drain on replacement inventory was significant, not to mention the CSEs' time and burden rate.

The Costs:

If a good unit is marked bad and sent back to the repair center, it causes a drain on inventory and costs, plus:

• Additional capital utilization by placing a replacement unit at the customer site
• NDF service fees (inhouse or external), plus time and expense to test and put a unit back into circulation
• Shipping and handling back to the repair center

There is also a significant inconvenience for customers when swapping out a unit.

The Challenge

StorageTek was originally a mainframe-oriented company with CSEs trained to look at MVS Logs. Once they went to Open Systems/SCSI/FiberChannel, they could no longer rely on host system tools to diagnose problems, and StorageTek CSEs did not have the tools and training to work in the Open Systems environment.

There were two challenges: first to quickly diagnose problems, and second, where necessary, to upgrade Firmware to current levels. Both were difficult without the right tools, especially with the range of drives in the field. In order to identify and fix problems, CSEs had to create dump trace tapes, record statistics, and ship the tapes back to the factory for diagnostics. This process took days and multiple trips to the customer site, meanwhile, the customer was out of service. To upgrade firmware, the CSE's had to create Firmware Upgrade Tapes for each drive type. With many drives in the library, it took a long time to do all of the drives, one at a time.

The Requirement:

StorageTek needed a tool that would enable CSEs to quickly and accurately diagnose equipment and software problems in the field, so they could get the customer back on-line, and the CSEs on to their next service call. They needed something that was sophisticated enough to test and diagnose StorageTek's wide range of drives, including SCSI and Fibre Channel. It had to determine with a high degree of certainty, whether it was an equipment problem, a problem with the media, or a down-level version of firmware. It also had to be quick and easy for the CSE to use. CSEs work on many different products and can not spend time learning new tools. It had to be intuitive.

So an easy- to- use Graphical User Interface (GUI) was definitely a requirement, with easy- to- read displays of test results that did not require a lot of interpretation. Pop-up Windows-Prompts were needed to remind the CSE to take different actions. It also had to be highly portable.

The Solution

About 9 years ago, StorageTek started working with SCSI Toolbox LLC, and today has a portable testing suite, deployed on a notebook computer that is used by over 800 CSEs in the field. The StorageTek SCSI/Fiber Channel Toolbox enables the CSE to do real-time testing and statistical analysis, and determine whether the unit is faulty, whether it can be fixed at the customer site, or should be sent back to the repair center for more extensive repair.

With the addition of a Magma PCI Expansion Box, the CSE is able to run native channel interfaces to the units. A SCSI LVD, HVD, and even a Fibre Channel Host Bus Adaptor can be driven from a laptop using the Magma PCI Bus Expansion. There are different tests for different products, small libraries, medium libraries, Quantum tape drives, StorageTek tape drives, and LTO tape drives. The different tests have been developed to identify failing units, Go/No-go testing, performance measuring, and Read/Write testing. The results are definitive and give feedback and direction on what is the problem and what is the fix. There is not a lot of room for mis-interpretation or error. If the drive is bad, they can send it back to the repair center, with a high degree of confidence that they are not taking a good asset out of circulation. More likely, they can diagnose the problem and fix it on-site.

The Benefits:

StorageTek's Toolbox testing suite allows StorageTek to better serve their customers. A StorageTek CSE can respond and resolve a customer problem quickly. The StorageTek SCSI/FC Toolbox is written specifically for the StorageTek CSE and the drives on which they work. Specific areas of concern can be interrogated, exercised, and displayed in easy-to-follow instructions for a resolution, which turns into a competitive advantage for StorageTek. Furthermore, the StorageTek SCSI/FC Toolbox tests have reduced the number of NDF units sent back for failure analysis. Units passing a Go/No-Go test in the field are not returned to the repair center, thus reducing NDF rates.

The Toolbox has also reduced:

• time CSEs must spend on the customers' site to diagnose and resolve problems
• the number of repeat visits to customer sites to resolve a problem
• the number of replacement units that must be taken out of inventory
• shipping and handling costs
• repair center service fees

The STK SCSI/FC Toolbox has become a tool that StorageTek CSEs rely on every day to do their job.

Extending the application

SCSIToolbox works with StorageTek engineers in advance of new product introduction, to ensure that as soon as products are in production and being shipped, the diagnostic tools with specific tests to service and support those devices, are available to the CSEs in the field. Whether requirements from Engineering, feedback from field CSEs, or SCSI Toolbox anticipating future needs, StorageTek and SCSI Toolbox, are constantly coming up with new ways to make testing faster, easier to use, and more accurate.

Rich Luif, is a Maintenance Development Engineer at StorageTek ( Louisville , CO ). Rich is responsible for supplying CSEs with the tools that enable them to diagnose problems and maintain equipment in the field.

www.storagetek.com

Michael Jones is CEO of SCSI Toolbox LLC ( Littleton , CO )

www.scsitoolbox.com

StorageTek Sidebar:

ABOUT STORAGETEK ®

Storage Technology Corporation (NYSE: STK) is a $2 billion global company that enables businesses, through its information lifecycle management strategy, to align the cost of storage with the value of information. The company's innovative storage solutions manage the complexity and growth of information, lower costs, improve efficiency and protect investments. For more information, visit www.storagetek.com , or call 1.800.275.4785 or 01.303.673.2800.

WORLD HEADQUARTERS

Storage Technology Corporation
One StorageTek Drive
Louisville , Colorado 80028 USA
1.800.877.9220 or 01.303.673.2800

SCSI Toolbox Sidebar

SCSI Toolbox develops custom software testing tools for storage manufacturers, OEMs, resellers and system integrators, including: design, development, manufacturing, quality inspection, and maintenance applications. SCSItoolbox was designed for ease of use, speed, is highly portable for field use, and is tailored to the specific needs of storage manufacturers and integrators. All processes and test sequences are created from a graphical user interface – no programming or scripting is required. A simple point-and-click interface allows complex sequences of tests to be assembled in minutes, and with many pre-defined test sequences included for Media applications, test lines can be up and running quickly. Time from set-up to a complete test process can be as little as 5 minutes.