Q

Home » Top Secret » Disasters with Shit Poor Programming

Disasters with Shit Poor Programming

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 44 other followers

I have finished up with my star field, a 3d background leveraging OpenGL, and with that have firsthand evidence of the shit poor coding that’s rife throughout the world right now when it comes to Windows Application programming.

During my last gig at Wells Fargo as a Senior Enterprise Architect, I had been called in to resolve a catastrophic problem occurring with close of business on Fridays, where a 60,000 user system would see the system slow to a crawl and frequently crash at 4:30 pm on Friday.

So with 60,000 people trying to get out of work on a Friday having to wait for this system to clear their transactions, sometimes their departure was delayed by as much as two hours as they often saw their transactions disappear from the system altogether.

As I investigated, the core of the system was a product called ‘Misys Financial Software’, which was developed by an Indian firm based – strangely enough – out of London.

Misys worked fine and dandy for a small size business, but as Wells took their foreign exchange from small to medium to large, Misys was scaling poorly. Horribly in fact. It was simply never meant for this volume of transactions.

I met numerous times with sales associates, engineers, and every level of risk, so I helped them traced the problem to something called Pub/Sub.

Now Misys had developed a WONDERFUL product for mid to large sized businesses, but it’s not uncommon for larger size businesses – and quite often nowadays even medium sized businesses – to obtain source code and/or to expand on the ‘open interfaces’ of software created by vendors such as Misys.

In Wells Fargo’s case, Wells had done a classic job of misplanned expansion. There’d been managers, directors, and executives who wanted to see information in different ways, which had seen a relatively small department in the foreign exchange have tons of financial resources thrown at it real quickly to scale, but this scaling was done without planning and leadership.

So while one internal group in Wells Fargo would ask for a change. Another group would as well. which eventually led to 600 people with direct access to the framework, 300 with database access, and 6 passwords shared between them all.

So in MiSys’s defense, Without a Database Administrator and with direct SQL embedded in a variety of different languages such as C++, Python, Java, and more, two shit poor programmers made disastrous changes to the SQL Databases that Misys relied on, Misys developed databases I might add, which when these ‘clogged arteries’ where struck by 60,000 users concurrently on close of business, well the whole thing acted like a row of dominoes.

But there was more to it than this.

I’d done some digging to learn – why had Misys given such ridiculous access to the users? I mean, when I developed Enterprise level solutions, I sure as hell made sure that I safeguarded my core databases to protect it from this…

There had to be more to it.

There was.

Now it was my responsibility to find a long term solution to this problem. So first, I advised that Wells Fargo immediately hire a high level Database Administrator – someone with a great deal of skill and expertise in optimizing and securing databases. The purpose for this was for segmentation purposes – to create team level accountability for work in specific areas. So if someone was hired on as a C++ coder, while they might have SQL Server coding background they could certainly assist the SQL Server DBAs, but the DBA was the one accountable for the database, as well as protecting and testing the code accessing the database.

The next step I had was reinforcing a rule – all SQL server access would be done through stored procedures. This made sure that the DBA could secure the database access points and reduce the chance of deadlocks and other issues with too many hands in the database.

As I quickly learned, putting these requirements in place too quickly would handicap all development efforts, something MiSys was well aware of, which is why Misys had provided access to the databases – as their ‘off the shelf’ solution wasn’t providing Wells with the capabilities it needed.

I started talking to the teams accessing the database, they showed me Misys’s original architecture.

It was .NET. And a relatively easy architecture to access the functionality that was caving the application in.

And herein lied the Achilles Heel of Misys’s solution.

This single access framework leveraged something they referred to as PUB/SUB.

or Publisher/Subscribe services written to handle distribution of transactions.

I spent the better part of that month on conference calls between MiSys London and India, and Wells Fargo employees across the nation, and learned that Misys’s services was NOT very scalable.

Specifically: This service that required the database work arounds ran on one server and one server alone, and Misys’s idea of scalability was to increase machine specifications. Need it to be faster? Increase processors and speed. Need more users? Increase memory. Need more data throughput? Increase network connections and disk drives.

But Wells had as much machine as could be thrown at it, and at a fraction of the users where I saw it collapse, Pub/Sub regularly crashed as a result. it plain and simply was not a viable option.

And they were lost on how to fix it.

Now I won’t get into how I helped Misys mitigate this problem, and how I helped Wells Fargo. I was paid shit for that gig and both these companies probably made billions off my changes.  But to be clear. there was shit coding by both Misys AND Wells Fargo which created the fiasco to begin with, which nearly collapsed the bank itself because – as we learned – there were traders taking advantage of this latency to Wells Fargo’s. disadvantage to the tune of $500 million a week.

That’s life and why I’m homeless, right? I undervalued my own services.

No matter. There’s a lesson learned about it.

What’s this have to do with my current endeavors.

Messaging.

So I am in the process of a complete overhaul of Windows 7, and converting it to something I am calling “Windows Holographic Edition”, the primary goal with this is to reprogram the developer’s mind to get used to working within a 3d environment, and to also provide some pretty cool off the shelf suite of tools for design and development of 3d game, art, and entertainment.

I just finished up with my Star field yesterday, when I had a thought.

The human eye tends to see between 24 to 30 frames per second. Some eyes faster, some eyes slower, that’s just a general average.

I’d been having problems with the Star Field and CPU usage – it would regular run in the threshold of 40 to 50% – and while it’s something I wanted to run all day, I immediately found myself disabling it because of the problems it caused with other applications.

So going back to the thought – I got to thinking…. My application is clean. Fast even. Small.

And there should not be any real problems with it.

So today I made a change to it, based on the concept that the human eye experiences 30 frames per second, to alter the code to redraw the 3d star field once every 33 milliseconds, or 30 times every second (1000.0/30.0) which should provide a pretty consistent experience  without hogging the processor.

I was delighted to see this change immediately reduced the processor load by a third.

But then I saw it.

The glitch.

And the entire background froze for a moment.

Another glitch.

And then I realized my problem.

Other people’s shit code.

Windows is a message based operating system. What this means is everything is designed like windows is a post office. Some messages may get sent to the post office (windows), which might get stored, or might do other things. But most messages are dispatched – sent to the respective addresses (windows) – dropping the metaphor – whether that’s Firefox, Internet Explorer, uTorrent, or it’s a media player like VLC.

In order for this system to work reliably, there’s an expectation that the applications developed will be developed with respect for this ‘message pump’ and message processing system.

But when applications don’t. After enough time with enough poorly coded applications.

You get glitches. Hiccups. And even crashes.

Now here’s the thing. The application I coded up should update the window – glitch free – every 30 milliseconds.

Either time’s moving different for me. And 30 milliseconds to the machine takes a full minute of time relative to my perspective.

Highly unlikely that’s what’s occurring.

OR.

There’s really rudely written applications out there which are demolishing my machine.

With the prevalence of popups. Adware. Device Drivers And Windows Services installed for any stupid reason because some developer gets a hairy bug up his or her ass to create one. Poorly handled thread based programming. Poor memory management. And more…

THIS is why I’m rewriting my own version of Windows.

Scrapping the explorer shell.

Deleting winint, svchost and anything else I can in the c:\Windows directory that adds to the sheer volume of concurrent processes running on my system (I currently have reduced my processes to 49, I’ve yet to understand what half these other proecsses are but regularly kill or disable processes which spike my CPU or memory usage)

The simple fact of the matter is. With 3d coding and OpenGL and a limited computer to begin with, my resources are precious to me.

The sooner I can roll my own web browser and delete firefox from my system the better. I enjoy it, don’t get me wrong, but like every other browser out nowadays, they all seem to care more about catching up to ridiculous standards than they do actually provide the customer benefit.

Same thing for newer versions of Windows.

They seem to care more about making a buck than they do actually providing me – the customer – benefit.

So AS I trim Windows 7 down to my base essentials.

As I rewrite my key components.

A 3d version of calculator, notepad, desktop, task manager…

I can’t wait to see if my system performance without the ‘off the shelf’ solutions work out best for me.

I can’t help but think there’s a lesson learned between messaging with Microsoft and Misys.

Other than maybe it’s best I control all my code for my own operating system, and maybe make that my final goal, to create my own 3d operating system from the ground up taking the lessons learned from retrofitting windows and apply it to a 3d Operating System I can sell and compete with Apple and Microsoft on.

Or maybe not sell it at all and just use it give it to friends should they want something different that actually works well with a tightly controlled design.

Hmm…

but what would I call it?


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Enter your email address to follow this blog and receive notifications of new posts by email.