You were hacked*: Infrastructure Audit

*Probably. But you haven’t noticed yet.
Your infrastructure is already compromised. So is ours. And, indeed, any that fundamentally interacts with the outside world. Rare exceptions might be specially guarded isolated nodes with air-gaps, where every terminal's USB ports are filled with epoxy, and an unfriendly security officer stands watch.

When it comes to normal business operations involving real people, prepare yourself for the inevitable: you will be attacked. Moreover, if all still seems well, you probably just haven't realized it yet. According to IBM research, it takes organizations an average of 187 days to discover they've been hacked. Every additional day a successful attack goes unnoticed increases the damage and the attacker’s control over the infrastructure. IBM's report also states that the average cost of data breaches in 2023 was $4.45 million. Yes, the numbers might be less staggering for smaller corporations, but the outcome will hardly be less painful for the business.

People Tend to Make Mistakes
The problem often lies in the simple fact that people are just people. They forget, make mistakes, easily confuse phishing emails with genuine ones, and generally behave in the least secure ways possible. It would be interesting to conduct a study on how many information security managers at large companies go prematurely grey.

The only way to mitigate these risks is to minimize human dependency. In our infrastructure audits, we always adhere to principles like “You can't leak a secret you don't know. It's hard to break something if you don't have the permission.” Building secure infrastructure is a lengthy process, but it goes much smoother and more transparently if you have a solid plan that describes it step-by-step. This often requires not only studying the client's infrastructure but also immersing ourselves in their business processes. This approach not only ensures security but also convenience.

For example, implementing HashiCorp Vault allows employees to perform complex actions requiring privileged access to secrets without actually revealing them. Everything just automagically works and is logged. There is a minimal chance of users causing password or key leaks because they do not have direct access to them.

You might also want to look into Teleport. It’s an excellent solution that enables you to implement a role-based access model in your company. They offer a great community version that allows you to roll out a role-based access model to company resources for free, record sessions, and keep extensive logs. Consider it if you’re looking for a PAM solution to start with.
Updates Need to be Automated
Another common issue we encounter is updating OS and system components. Yes, a server that hasn’t been rebooted for years and runs without updates is a relatively simple solution. Unfortunately, it quickly becomes a delicate snowflake that engineers are afraid to touch because no one knows if it will survive an update or reboot.

Many of these problems are solved by IaC—Infrastructure-as-Code. If you’ve designed and automated all processes well, the risk of human error is minimal. According to Verizon, it takes an average of 55 days from a critical vulnerability discovery to patch application in an organization. If that’s still how things work for you, you definitely need to audit your processes. It’s usually during this time lag that the most successful attacks occur.
In Short
  1. Everyone needs an audit. A fresh perspective from qualified engineers can spot problems that you’ve grown accustomed to overlooking.
  2. Everyone will be attacked. It's almost axiomatic. The only real defense is to make the cost of an attack on you higher than the potential gain from the breach.
  3. Just because you don't see the consequences of an attack doesn't mean all is well. Often, a hack is discovered only six months after it occurs.
  4. People make mistakes. They are almost always the weakest link in the chain. Minimize the secrets they can access and limit their permissions to the bare essentials. Strongly consider looking into HashiCorp Vault and Teleport. If you don't know how to set these up, come to us, we can help.
  5. Automate the updating process. If you are updating the OS manually, you have a problem. If you aren’t checking your images for vulnerabilities with a well-configured CI, you have a problem.
  6. An audit doesn’t mean you need to burn everything down and start over correctly. A properly formulated plan will describe the digital transformation step-by-step and at a pace that’s comfortable for the business.
If you need help, write to us. We will take care of your infrastructure.
Gumeniuk Ivan
DevOps Engineer