When working in healthcare IT support, there are certain phrases that always come up in our meetings with clients:  

“We’ll deal with it later.”  

“It’s probably fine.”  

“We’ve never had an issue before.”  

Even after working for so long in this industry, hearing any of these still sends chills down our spine EVERY. SINGLE. TIME. We’ve learned that these phrases rarely show up in companies where everything is running smoothly.

Table of Contents

The Help Desk Guy

The company was a growing healthcare organization with more than 200 employees and multiple locations. As a healthcare provider, they handled extremely sensitive patient information, relied heavily on technology, and needed their systems running without a hitch. They approached us for our extensive experience in healthcare IT support and biotech. The idea was that we’d assist in securing PHI (Protected Health Information) and maintaining the infrastructure.  

We heard the usual phrases, which at this point was just expected, but there was something else they said in passing that set our mental alarms off:   

“We have a Help Desk guy.”  

It was our first red flag, the first sign something was wrong, and we knew that, but we didn’t yet fully grasp the weight of that phrase at the time.  

You may be thinking: every company has a healthcare IT support department, and within that department, a Help Desk guy, right? What is the issue? Well, the problem here was that they quite literally had a singular “Help Desk guy”, and only that Help Desk guy. That’s it. No department, no team, just Josh, the “Help Desk guy”.

One-Man IT Department

Now, this isn’t a story about blaming Josh. Josh wasn’t even lazy or incompetent. He was simply expected to be an entire IT department by himself. He was meant to handle user support, systems administration, cybersecurity, Microsoft 365, networking, compliance concerns, and every other technical issue that appeared throughout the company.  

Josh could handle many of the day-to-day issues, but the organization had a major gap: there was nobody above him in the IT department to whom he could escalate. No senior engineer to deal with complicated problems, no IT manager, no CTO providing direction. Not a single soul with more experience and technical know-how that could be trusted to solve the problem when Josh was out of his depth.  

And as it turns out, that’s why they hired Network Antics and why they wanted healthcare IT support.

Naturally, the first step in any managed healthcare IT support relationship is to check out the client’s infrastructure and get a better understanding of what we’re working with, so we headed on over to their headquarters. What we found surprised us, in a bad way. 

Welcome to the Jungle

When you walk into a client’s environment for the first time, you never really know what you’re going to find. Sometimes everything is clean, organized, documented, and properly maintained. Other times, you open the server room and immediately understand why they’ve been having problems.  

This was the latter.  

The infrastructure was old. There were cables everywhere, a general lack of organization that made it difficult to tell what was doing what, and equipment that had clearly been gathering dust for years. That’s not necessarily a problem by itself. Old equipment can still work well if it’s properly maintained, secured, monitored, and replaced when necessary.  

The problem was that this environment was not being properly maintained, secured, monitored, or replaced. It felt more like the abandoned ruins of an ancient civilization.  

We started documenting what we found, taking pictures, looking at the servers and switches, and getting a better understanding of how everything was put together. And the further we looked, the more obvious it became they hadn’t even touched their equipment in years. Their server infrastructure was from ten years prior. Their operational system was even older.

The Backup Wasn’t Backing Up  

Eventually, we got to one of the most important questions you can ask when looking at a business’s IT environment:  

“You have a working backup, right?”  

After a long, awkward silence, Josh explained their backup situation. Their Veeam backup had stopped working approximately six months earlier. Six whole months. When the previous help desk left, and Josh took over, there was no proper standard operating procedure in place to ensure the backup was properly handed off and maintained.  

So, for half a year, the company had been operating under the assumption that their data was protected when their backup wasn’t actually doing its job.  

A backup system isn’t something you set up once and then forget about. It needs to be monitored, tested, maintained, and regularly verified. Otherwise, you may as well not even have a backup. And in a healthcare environment containing sensitive patient information, that’s a particularly dangerous thing to have, or rather, not have.

Point of Escalation  

The healthcare organization was put on a retainer contract. We became concerned about the possibility of “invoice fatigue” due to the potential of a catastrophic event taking place given all the red flags. The main issue was that they wanted our services to be reactive, rather than proactive. We wanted to act and fix the glaring issues before anything bad could happen, but they wanted to save money in the short term and declined it. They didn’t consider that in the long term, if something severe did happen, it’d cost them infinitely more. 

We were hired on as a point of escalation for issues the help desk person had limited bandwidth or simply did not know how to resolve. While alerting of security concerns and aging infrastructure, it was not our job beyond resolving what ticket was passed on to our team. 

Reactive vs. Proactive Managed Healthcare IT Support

One of the advantages of working with a managed healthcare IT support provider is that you’re not just hiring one person. You’re getting access to an entire team, along with the tools, systems, processes, and experience behind that team.  

Our managed IT stack includes the systems we use to monitor environments, track tickets, manage users, document infrastructure, identify problems, and keep everything organized. It’s a great asset to have when you make full use of it, which unfortunately was not the case here.  

The healthcare company didn’t want us to take the proactive managed IT approach, which means fixing things, improving standard operating procedures and closing any gaps before they represent a security risk. Addressing things before they become serious problems tends to, unsurprisingly, avoid those serious problems.  

They preferred that we handle things reactively, though.  

Something broke.  

Someone opened a ticket.  

Someone fixed it.  

Everyone moves on.  

Repeat.  

But the underlying issues that keep causing things to break all the time aren’t getting fixed.  

It’s like having frequent headaches, taking headache medication, but never going to the doctor to understand what’s wrong. Sure, you can go years without any problems, but you might also end up discovering you have something really serious, and because you took so long to do something about it, it might be too late.  

A Storm Brews on the Horizon

Over the next couple of months while working together, we tried warning them and helping fix their issues, but did not budge at all.  

Needless to say, falling victim to hackers and ransomware is one of the worst things that can happen to any business, but when you’re in the health industry in specific, that data not only is a much more valuable target, but losing it is also a much bigger headache. 

Despite the seriousness of the situation, they did not seem to give the matter the importance it deserved. They believed that because nothing bad had ever happened before, surely it couldn’t possibly happen in the future, right? Well, with our experience in healthcare IT support, we knew it wasn’t a matter of if, but when. And as it turns out, that “when” was sooner than even we could’ve expected… 

Read the next part of this adventure:  

Adventures in IT Part 2: Healthcare Cybersecurity Risks – The Breach

This blog post is the first in our new series called “Adventures in IT”, in which we relive some of the juicier and more impactful real-life stories we’ve experienced in the managed IT business that left a lasting impression on us. This is so that you, the reader, may learn not just from your own mistakes, but also from the mistakes of others.