Tools to Detect Profile System Performance Issues
Profile projects that introduce change on a large scale, such as version upgrades and new product rollouts, invariably create new risks to Profile system performance. This blog offers a high-level view of the approach and tools used to test and detect issues before they cause catastrophes in Profile production systems.
The Approach
In Profile, change often begets performance issues. Profile pre-production test environments provide an institution a platform to identify, define and resolve issues before they are released into a live production environment. Before an issue can be resolved, it has to be identified. This requires testing and monitoring areas where system resources are most burdened. Two key processing areas to test are the dayend event (which includes batch processing), and online, real-time transaction processing.
To start, test parameters need to be defined based on current live production scenarios, which includes consideration for items such as the duration of the tests; the number of test case executions per minute; the ‘think’ time between calls and the number of concurrent threads running per test case. Test cases are then created and include the examination of Profile message logs; the extraction of distinct MRPC and SQL calls; the creation of meta files for each; the generation of input files from each meta file; and the execution of the load generator for online transactions. To conduct thorough and efficient testing, tools are required. Experience and knowledge are also needed to interpret and analyze the gathered information correctly.
Tools Used
Let’s look at the example of testing dayend processing. In testing, the elapsed run time for each job is used to determine the overall length of time it takes to run a full dayend. The jobs with the longest run-times become the focus of the performance evaluation. In addition, CPU usage, memory utilization, file I/O and database contention are reviewed to determine if there is an opportunity to optimize the code and reduce the overall runtime.
Continuing with the dayend processing example, a bank determines that its upgraded Profile version must run and complete its dayend within the same period of time as the system’s prior version. However, dayend jobs running in the post-upgrade test environment are showing the entire dayend to run several hours past target.
The QUEUEHD table is used to identify the longest running jobs during dayend. The GT.M operator log is used to identify potential database contention that may be impacting performance. This information highlights areas where additional scrutiny and focus is most likely needed. For example, the log file might point to the need to investigate the code surrounding a multi-threaded batch process that continues to trigger transaction processing restarts (TP restarts) resulting in increased demand on resources.
Additionally, to monitor all system resource usage while dayend runs, an AIX/Linux utility called nmon is used. The tool graphically depicts CPU usage, memory usage and file I/O over time, making it easy to spotlight spikes in the usage of each. In conjunction with the GT.M event logs, dayend jobs running during spikes in system usage can be identified. Using these tools together significantly narrows search effort and more quickly points to areas where there is a higher probability of issues that can adversely affect the dayend-processing window.
A second key area where system resources come under heavy burden is online, real-time transaction processing. To test in this area, MGP consultants use a ‘customized-for-Profile’ version of Jmeter, an Apache open source Java application used in real-time transaction load testing. JMeter sends test transactions to Profile and monitors MRPC (remote procedure calls) responses, SQL response times and response times for Profile’s Web interface for the back-office. The nmon utility is also used to measure CPU usage, memory usage and file I/O for online response times. And, just as with dayend processing, the GT.M event logs are used to monitor database contention during online transaction processing.
This chart was not generated from a Profile test, but rather it’s a screenshot taken from Apache’s JMeter website. It is shown here as an illustration of how easy it is to see the peaks and valleys of various test scenarios in conjunction with one another. JMeter’s filter function allows for the custom selection of test scenarios. When this tool is used in combination with others, along with a healthy dose of Profile system performance expertise, areas of concern can be identified quickly.
In the examples, the goal was not to solve the specific issues adversely affecting system performance, but rather it was to identify locations that warranted closer inspection. JMeter, AIX/Linux nmon and GT.M event log are part of a basic diagnostic set Profile tools that help identify and locate areas of high concern. To further diagnose, pinpoint and resolve performance issues, Mozaic Group Partners consultants have developed additional proprietary tools and utilities, which they use in conjunction with their Profile experience and expertise.
In addition to monitoring and testing performance for Profile projects such as version upgrades and new product launches, increasingly, banks are testing to understand the impact future growth has on system performance. Through the use of a future test environment and the tools described above, a bank can stress tests its system against its growth projections. The information gained can be used as a basis for future system resource budgeting.
If you have questions and would like to talk about how your institution could benefit from the knowledge and experience MGP consultants have with Profile performance monitoring and testing, please contact us. We are always happy to help.
About Mozaic Group Partners
We are the leading, independent, global provider of Profile banking application services. We partner with Profile institutions to develop, implement, fix, enhance and protect the investment your financial institution has made in its banking system. With more than 275 client engagements in 18 countries and counting, we have helped more institutions solve their Profile issues than any other services provider on the planet.
###
If you found this news article helpful, please share it with someone who might also benefit from its content.
