PKO Bank Polski Nearly Doubles Performance of Its Alnova Central Banking System With Hitachi Storage Solutions

Overview

Challenge

• To grow its business, Online Data Cloud needed a scalable storage platform that could deliver high performance and low latency for the cloud services it provides to its clients.

Solution

• The largest bank in Poland was challenged with increasingly degraded performance of its mainframe-based Alnova transaction processing environment. During peak hours, as well as during the end-of-day and end-of-month batch processing runs, the storage systems reached their performance limits. As a remedy, the bank chose to implement Hitachi data storage. In the race for this prestigious contract, Hitachi Virtual Storage Platform (VSP) successfully competed against mass storage solutions from 3 other vendors.

Outcome

• Processing performance nearly doubled.
• Data recording delays kept below the threshold.
• Precise performance and usage analyses enable easier planning and problem diagnosis.
• Capacity to support three years of growth without further upgrades

Challenge

Keep Transaction Performance

PKO Bank Polski is Poland’s largest retail bank. It exceeds PLN 200 billion in assets and operates over 6 million current accounts of individual clients. The bank has nearly 1,200 branches and cooperates with more than 1,000 3rd-party agents. It also administers a network of over 3,000 ATMs. PKO Bank Polski is one of the leaders of Polish online banking. Thanks to the iKO, a mobile payment service that it offers to cover all payment situations, it is now the undisputed innovator in the mobile banking segment.

PKO Bank Polski uses Alnova, an advanced transaction platform running in an IBM^® z/OS^® (mainframe) operating environment, which also includes a large mass storage environment. In peak times, the platform processes nearly 2,000 transactions per second. The performance of transaction processing is critical to the bank’s operations and depends primarily on transaction-delay times. Multiple specialist IT and business teams oversee the system to ensure it is stable and efficient.

“Our job is to meet the quality of service expectations of our clients. When using online and mobile services, such as Inteligo, iPKO or iKO, clients expect the services to be very responsive,” explains Daniel Sosiński, IT infrastructure director of PKO Bank Polski. “The branches need to work efficiently, too. In practice, even small delays may slow down the whole system, adding to service times. The speed of transaction processing is not only important in practical terms. It also affects the clients’ trust and the bank’s ability to timely file statutory reporting,” he says.

The new mainframe servers implemented in 2011 improved Alnova’s performance, but the needs were still on the rise. Performance of transaction processing does not depend merely on server platforms. Large databases do not fit into servers’ memory and need to be stored on mass storage systems. Therefore, the speed of transaction as well as batch processing, such as report generation at the end of a day or month, depends heavily on storage performance

“We had a top-class disk system with an enormous cache memory. Despite this, on mandatory payment dates and during endof-day and end-of month reporting loads the processors of storage controllers reached 100% of their capabilities,” recalls Maciej Księżycki, senior administrator from the Mass Storage Team in the IT Infrastructure Department of PKO Bank Polski. “We had to do something, as our plans assumed moving about 600,000 Inteligo accounts to Alnova. We didn’t know that then, but we were soon to take over Nordea Bank Polska. The bank wanted to replace its mass storage system, but we knew such operation could not happen overnight. Months were needed for laborious analyses and preparations, and a few more months for tests,” Księżycki notes.

Solution

Hitachi Virtual Storage Platform

The bank sent out requests for proposals to leading mass storage integrators, with very high performance requirements and numerous extra conditions. Requirements included the provider’s consent to make the offered systems available in advance to enable the bank to verify the parameters declared in the bid and to run comprehensive tests, primarily performance-related ones. The delivery of the systems did not mean winning the contract. It was decided that if the solution selected by means of comparing the bids failed to achieve the declared parameters, the hardware would be returned to the vendor with no extra costs for the bank. In such case, the hardware of the next provider on the list was to be tested.

The bank was perfectly aware which storage parameters were critical. The detailed requirements were prepared by a team of the bank’s experts responsible for applications, databases, networks, servers and mass storage.

“After a few months of tests of applications, system environments and mass storage, which preceded the publication of the request for proposals, we prepared a very detailed load profile for the whole installation. Quite a big team worked on it, about 10 people in total. Only after those examinations, during which we were sometimes analyzing single transactions, could we thoroughly define the actual functional and performance requirements for the new mass storage environment,” reports Urszula Michnowska-Czyż, leader of the Mass Storage Team in the IT Infrastructure Department of PKO Bank Polski.

Within those requirements, performance was key, as the bank wanted every assurance that the mass storage environment would add not more than 2.5 milliseconds to the total transaction processing time. Transaction processing delays exceeding 3 milliseconds were identified among the major causes of the slow-downs of the entire environment. As an additional condition, the bank wanted this delay threshold kept even with synchronous replication to the backup data center enabled.

“The winner of the initial stage of the procedure was Hitachi, offering VSP (Virtual Storage Platform) systems,” reports Księżycki. For test purposes, 2 VSPs were installed in the main location, with 90TB of raw disk capacity each. The resources on each storage system were partitioned to form 2 logical domains, 45TB each, in order to make local copies. An identical installation of 2 more storage systems was started up at the backup center. The distance between the locations was only 20 km, but for the tests they were connected with an optical fibre, 50 km long. “By this, we wanted to verify empirically if keeping the delay below the desired threshold was still possible with the backup site located further away than the one we use currently,” Księżycki says.

The testers installed on each VSP: 2 pairs of controllers with IBM FICON^® ports for mainframe servers, 2 pairs of caching controllers (total cache capacity of 160GB), 2 pairs of disk controllers, 2 pairs of storage system managing controllers, and a pair of controllers with Fibre Channel ports for remote replication. To test the storage systems in situations as close as possible to their prospective live environment, the bank’s engineers “recorded” about 3 hours of extreme-load work of its production environment. To this end, they used the Hiperstation application by Compuware, which had been well proven for similar purposes before. The recorded load was reproduced in the testing environment in several phases, with various configuration parameters of the mass storage environment, server platform and network.

“Over several months, we tested, step by step, every parameter included in the Hitachi proposal. We took the declared values very literally and we were very meticulous,” recalls Księżycki. “In addition, every testing scenario was run for different variants in terms of configuration and resources available in each layer of the environment, for different volumes of jobs being executed simultaneously, for a variety of simulated problems, etc. All tests were planned and documented so we could repeat them or compare the results when needed.”

Outcome

Nearly Doubled Performance, More Capacity

PKO Bank Polski did not have to test other mass storage systems. The test results for the Hitachi solutions were more than satisfying. “The Hitachi VSP storage systems came out excellently during the tests. All was as declared. In some scenarios, they considerably surpassed our performance requirements. And, what is more, the test outcomes were later fully verified in the production environment,” says Księżycki. ‘With the new storage systems, the longest batch processing takes about 7-8 hours, as compared to 10 hours before the change. The servers have not been replaced, and this clearly shows the extent to which the performance of application environments depends on mass storage systems,” he emphasizes.

Hitachi won because of excellent storage performance combined with appealing terms and conditions, and not only because of the price. For instance, an option for post-warranty technical support was a very appealing element of the proposal submitted by Hitachi Data Systems, and an alluring benefit for PKO Bank Polski. After 3 years of use, storage systems are usually still fully operational. From the technical perspective, the client may keep using them as before or for new purposes, that is, for a testing environment or for analytics or reporting applications.

“The option for post-warranty technical support is beneficial as it protects our investment. Our experience shows that loads and data volumes are mainly shaped by diverse market and business factors, and not our plans. If data and transaction volumes grow slowly, the support option will allow us to postpone the purchase of a new environment. If the growth rate is higher, we will certainly find a useful job at the bank for the storage systems just a few years old. In such a big organization, there are always some data processing needs which can benefit from migrating them to a top-class storage system,” suggests Michnowska-Czyż.

The implementation at PKO Bank Polski has also demonstrated that it is completely useless to assess mass storage systems only on a basis of numbers. “At each of our 2 data processing centers, we replaced one top-class storage system utilizing 512GB cache with 2 Hitachi systems with 160GB cache each. So, the caches in the 2 Hitachi devices together only slightly exceed half of the cache memory of the previous array,” says Księżycki. “Nonetheless, delays are now significantly reduced and performance is considerably higher. Performance is more than hardware: It’s also data management algorithms, and those from Hitachi seem really smart.”

The transaction processing environment at PKO Bank Polski is now stable and efficient, and the same applies to batch processing. Administrators can precisely analyze performance statistics on an ongoing basis. “I am especially impressed by Hitachi Tuning Manager, a software package provided together with the storage systems. With this tool, I can monitor the status of the whole system for the last 24 hours at one-minute intervals. Data collection for reporting purposes is fully automated and reports are a good starting point for detailed analyses,” says Księżycki.

PKO Bank Polski Innovates With Information

• Through empirical tests of performance and scalability of the transaction management infrastructure, the bank was able to identify more precisely the load limits of its transactional environment as a whole.
• Analysis of various operational scenarios of the transactional environment with unusual configurations and sequences of events enabled the bank to improve risk analysis and contingency procedures.
• Reliable assessment of operating and investment plans ensures continuity of business and a safe transactional environment for the bank.