Got Gaps In Your Profile Services?
Profile processing issues come in all sizes, have multiple origins, and cause various degrees of alarm. An issue can start small and remain undetected until its cumulative effect exposes itself. An analysis mistake in a project’s design or a single line of inefficiently written code can lead to system-numbing stresses and to an eruption of larger issues.
This isn’t news to most of the people reading this blog. However, it’s not what most client institutions know about Profile, it’s what they don’t know – the hidden and unanticipated problems that create the issues. This circumstance is known as the “Profile services gap.”
Processing problems can arise on any Profile project, but they are more likely to happen with projects that deal with:
- application framework,
- system performance,
- business and application analysis,
- third-party integrations,
- custom development, and
- version upgrades.
Service gaps can be directly linked to a lack of advanced Profile coding techniques and development experience, insufficient training, and a lack analytical expertise as it pertains to the application’s functions, features and customization.
Our job at Mozaic Group Partners is to (1) help Profile client institutions recognize their service gaps, (2) use our experience and expertise to correct issues and reduce operational risk, and (3) supply the training and mentoring to help the institution expand its Profile expertise.
In this blog, we highlight a few examples of issues that resulted from services gaps at Profile client institutions and how Mozaic Group Partners solved those issues.
Bringing Order to Chaos
Our first example happened at two different Profile institutions, and we suspect others have unknowingly encountered it as well.
An institution created a new product and supported it with strong marketing. At launch, the public’s positive reaction to the new offering generated a response that exceeded institutional expectations. However, as customers rushed to open new accounts online, the number of requests sent to Profile to generate new account numbers grew to 100 per second. Profile runs fast, but the volume overwhelmed the system and created a processing bottleneck. Profile, given the way it was configured, was unable to assign new account numbers fast enough to satisfy the demand. Much to the bank’s embarrassment, some would-be customers received online messages notifying them that the bank was not able to open their accounts and to “try again later.” The bank called Mozaic Group Partners to solve the predicament.
What happened? The strain of multiple, concurrent account creation processes trying to simultaneously read/write to the table responsible for assigning account
numbers resulted in the same account number being assigned to multiple concurrent account creation processes. With so many processes attempting to secure an account number from the table at the same time, some processes were grabbing account numbers that were already taken. At the end of the creation process, when Profile tried to file the account, an account using the same account number already existed and the database would throw a “duplicate account number” error. This error caused the creation to fail. In this example, the end result saw approximately 30 new account requests fail out of every 100 attempted.
The issue needed a code fix that would perform as a traffic cop. The answer was to install an error trap that forced a TP restart as soon as a process grabbed an account number that had already been assigned. And, instead of starting that process from the very beginning, the restart would begin closer to the point in the process where it needed to grab an account number from the table. Additionally, if the process was not able to secure a valid account number after three TP restarts, instead of failing, the process would invoke GT.M’s critical section which would suspend all other processes until the affected process received a valid account number. The modifications forced the completion of each transaction and guaranteed that all processes received a unique account number. The result solved the problem and, most importantly, was transparent to the customer.
Up until the time of the new product’s launch, the institution had not encountered issues with Profile assigning new account numbers. However, the pre-product launch analysis and testing did not consider account creation volumes beyond what the bank had normally experienced. Had the effect of larger account opening volumes been considered up front, testing could have caught the deficiency pre-launch and the code could have been modified to process high-volume account creation. Advanced application training and coding techniques helped the bank close its Profile services gap on high-volume account creation.
Ending Never-ending Dayends
What institution doesn’t want to make its dayend process run as efficiently as possible? Institutions want to close one processing day and to open the next in the most efficient and quickest way possible. Our second example concerns dayends that take excessive time to complete—some taking eight, nine and even 12 hours.
In essence, Profile runs a daily race during which performance degradation can happen gradually or suddenly. It can happen over time as a dayend gathers clutter or the database increases in size, or it can happen suddenly following a version upgrade. In one instance, an upgrade to a Profile 7.x version pushed a bank’s end of day processing to an unimaginable 36 hours! With that particular institution, Mozaic Group Partners was engaged to review, troubleshoot and resolve the issue. We found discrepancies and defects in the custom code that was upgraded and in the framework itself. In turn, this adversely affected the handling of the institution’s
custom code and significantly increased the processing overhead.
Improving dayend performance could require an adjustment to a process in the critical path, or a re-examination of the effects of custom code, or an evaluation of the length of time it takes to run a report. When Profile client institutions call us to fix a lagging dayend performance, our analysis typically finds an accumulation of processes with performance-zapping issues that have manifested into larger concerns.
Regardless, we recommend institutions review their dayend processes thoroughly every six to 12 months or at the first sign of performance degradation. This is a service gap that institutions can remedy with training as it applies to application knowledge and better code development skills.
Reports are typically run as an event separate from critical path processing (or on a separate reporting instance), but they still consume system resources and need to be run on a timely basis. For example, at one engagement, we found a report that the institution had run every day for two years. Every day, that single report took 12 minutes to run. After we flagged the report, a quick analysis showed that to accommodate the report’s data sort, the report definition was set up to go through every account in the system. In a test database with 10,000 accounts, that report runs in two seconds, but in a production database with tens of millions of accounts, that report runs in 12 minutes. Our fix was to use a different technique to build the sort in a more efficient manner. A report that took 12 minutes to run now runs in less than 15 seconds. Minutes add up quickly, which makes it important to maximize efficiencies in every process during dayend wherever they can be found.
Taming of the Batch
In our third example issue, we deal with one aspect of managing batch efficiency. Batches are the workhorses of dayend processing and if not configured and managed properly they can extract a weighty processing toll on system resources. The ability to create consistently efficient batches is one of the most common services gaps found at Profile client institutions.
In this example, we examined an overnight batch that performed a very system-intensive process for a small number of accounts (relative to the institution’s very large database). The processing work involved per account was taxing and took an inordinate amount of time to process each record in the batch.
Profile batches use multithreading to speed up their processing by splitting the batch into smaller tasks and executing those tasks in concurrent threads using the availability of multiple CPUs. Profile's batch system selects a list of records to process and feeds each "chunk" of records to individual processing threads. When working with a large number of records, Profile's 32,000-character record size "chunk" works well. However, in this case, with only a small number of records to process, all of the accounts were fitting into a single "chunk", which meant the batch was running as a single thread and could not take advantage of the more efficient multi-threaded functionality. To make this happen, we reduced the batch’s message buffer size from 32,000 characters to a much smaller number. The reduction in the message buffer created smaller “chunks,” and allowed the batch to capitalize on its multithreading capability.
Solve Your Profile Services Gaps and Protect Your Investment
For many Profile client institutions, services gaps are real and so are the operational risks they pose. Despite best efforts and intentions, services gaps do not improve on their own. Internal trial and error attempts are resource intense, costly and might not be supported with enough Profile knowledge and expertise to set them on a path of success. Considering the investment institutions have made in Profile, it makes sense to give internal resources the advanced knowledge and training to 
protect that investment.
In the world of welding, inexperience, inadequate training and lack of expertise put entire projects and investments at high risk. Welders with advanced knowledge, experience, and demonstrated expert technique are sought the world over because they produce high-quality welds, which reduce risk and ensure success. The people at Mozaic Group Partners are the world’s premier welding equivalent for Profile. Not only are we ready to solve your immediate issues, but our Profile experts can augment your internal projects and mentor and train your resources. We are experts with advanced Profile knowledge and coding techniques that can help you with framework, system performance, business and application analysis, third-party integrations, custom development and version upgrades. If you have questions and would like to talk about how your institution could benefit from Profile mentoring, advanced training, or a code workshop, contact us. We are happy to help.
About Mozaic Group Partners
We are the leading, independent, global provider of Profile banking application services. We partner with Profile institutions to develop, implement, fix, enhance and protect the investment your financial institution has made in its banking system. With more than 275 client engagements in 18 countries and counting, we have helped more institutions solve their Profile issues than any other services provider on the planet.
###
If you found this news article helpful, please share it with someone who might also benefit from its content.
