Breaking the glass: Mastering BreakGlass Techniques for Emergency Access

Emergency situations call for emergency preparedness. The term “breakglass access” derives from the world of emergency alarms (such as fire alarms) that are protected by “break glass” stations, where once the alarm is activated it cannot be “turned off” without replacing a part of the station. Sometimes the fire alarm has a glass or plastic insert that has to be replaced after the alarm is activated. In any case, a responder is going to immediately recognize that the alarm has been pulled.

Table Of Contents

A. "Breakglass access" in the digital age
B. Retain role based security – Emergency access to particular levels of “the stack”
C. Use cases: When emergency access is required
D. Emergency access suggestions
E. Concluding remarks
F. References

A. “Breakglass access” in the digital age

In computing, “break glass” is the procedure to access a system that bypasses normal security controls during critical emergency situations. Break glass procedures rely on pre-staged emergency user accounts that are documented, tested, and managed. For example, a “break glass” admin account may be created for situations when network based authentication/authorization services (such as Active Directory) have become unavailable. The break glass accounts should be made in a way that they rely on (1) the user and (2) the target system, with very little tertiary system involvement.

Of course, in all break glass situations, be aware that the break glass accounts can also be weaponized by threat actors. Since the break glass accounts bypass potential mitigation steps, a threat actor may be able to use them. For example, break glass accounts rarely enable conditional access policies such as MFA. Without a second factor to security, a threat actor has easier access to the systems that are being protected.

It is also important to note that “break glass” access is not always a “break glass” account. Break glass access might be a method or procedure. For example,

Break glass in a data center might mean that there are methods to boot the affected system in a Safe Mode container that provides properly authenticated access
Break glass in a cloud environment might mean that there are procedures available to call the service provider and have a new account created.

B. Retain role based security – Emergency access to particular levels of “the stack”

Software is a many faceted beast, including infrastructure (networks & servers), platforms (operating systems), and software (reference AAS sisters). Emergency special access rights need to be configured for all three layers of the beast.

For example, let’s say you have a website built on WordPress deployed on a web hosting server. There are several break glass opportunities and scenarios. To outline a few, there are (1) the website, for example, where new articles are created; (2) the WordPress deployment, for example, where new users are created; and (3) the web hosting login, where a new WordPress might be created. There are of course many others.

But there is no reason to get carried away with break glass accounts. As a reasonable starting point, understand what each break glass account is capable of doing. Do you really need this many break glass accounts? Probably not if you control the entire stack.

If access to the website account is lost, the normal WordPress Admin account authorizations can be used to change the website account password.
If access to the WordPress Admin account is lost, a new account can be created by the normal web hosting login.
If access to the web host is lost, a reasonable break glass procedure might be to call the hosting provider and have the access credentials reset.

C. Use cases: When emergency access is required

To better understand how to protect systems with break glass access, let’s explore why emergency access may be required. To name a few, emergency access may be required in the following situations:

Cyber attack (insider or external) has deleted or removed access to all accounts. In this way, the system is unavailable by all methods other than break glass.
Accounts are federated, and the identity provider is not available. For example, if access to AD has been compromised by way of a cyber attack, or a network outage has prevented access to AD, the system is unavailable by all methods other than break glass.
Multi factor is enabled on all accounts, and the Multi factor grid is not accessible or has become compromised. For example, in a global phone outage (text based MFA), or if an MFA app provider has become compromised. In this situation, the system is unavailable by all methods other than break glass.

Remembering that break glass access can also be weaponized by a threat actor. It is best to restrict the number of methods to gain access, to reduce the vulnerability exposures.

D. Emergency access suggestions

Break glass access is typically either

by way of system access procedures, for example, console access;
by way of contacting a provider company that has access (for example, in a cloud hosted environment);
by way of an account.

In any of the scenarios, the process should be documented and well tested. You don’t want to try to “figure it out” during a real outage that is affecting your users and customers.

Here are suggestions for emergency access:

Top five criteria for all emergency access methods

Fail proof – it has to work 100% of the time
Sufficiently privileged – in order to recover from every situation
Perpetual – not subject to lockout under any circumstance. Cannot be deleted, expired, nor deactivated, so that if a malicious user gains access to the system, the malicious user cannot execute a Denial of Service to the Break Glass account.
Not used for any access other than absolute emergencies – these are not daily access accounts
Regularly tested – triggered by time (say every 90 days), upgrades, updates, new break glass users, terminated break glass users

Additional criteria for emergency access

Simple – since the accompanying emergent situations is already increasing stress levels
Audited – with no ability to destroy audit trails, so that a “break glass” event is evident to observers
Protected – access methods should be stored in a manner in which if the method is accessed, the access is easily identified. For example, if break glass account, store the credentials in an envelope in a locked firesafe where the envelope itself has to be destroyed in order to access the credentials. In this way, anyone who has access can identify if the account information has been accessed.
Monitored – so that if the method is used, every user becomes immediately aware. For example, every admin is immediately notified that the break glass process has been invoked. Keep in mind if an adversary has gained admin access and admin notification occurs, the adversary will then immediately be notified that Break Glass has occurred.
Minimum necessary privilege to recover – for example, the ability to create and manage Admin accounts, where then the admin account can be used for the rest of the recovery process.. Remember, Break Glass is to regain access. The person who logs into the Break Glass account is not likely the person who manages daily access to the system. In a large environment, the Break Glass action is going to be used to establish a “fix beachhead” that is then used to regain global access for multiple other users.
Protected against single person insider threats – for example, requiring more than one person to gain access
Not assigned to an individual – since emergency access is to recover from an emergency, and the individual may be a contributing reason for the emergency (an insider threat bad actor)
Procedures kept current for any new versions or deployments of infrastructure, platforms, or software
Does not require reset, so that if part way through recovery another situation is encountered, the same break glass method can be used
Intentional – to protect against “accidental break glass”

Special considerations for “break glass” accounts

Not multi factor – because multi factor may be a contributing reason for emergency access
Local account – not relying on any centralized authentication or authorization services
Username/Password stored in a container where access is easy to identify and requires “new glass” (such as an envelope) to reset, that is, cannot be easily reversed.
Explicitly excluded from automated cleanup and lockout – cannot be locked out, ever
Explicitly excluded from lockout due to failed passwords – since an adversary could simply DOS the account to lockout break glass access during an attack
Access passwords or password locations changed when staff changes
Bonus: Password separated into two or three parts stored separately, with potentially different people having access to different parts of the password. Remember, breaking a password into separate pieces reduces the cryptographic complexity of the password. For example, if a 12 character password is broken into two 6 character segments, the resulting security is only that of a six character password. If an adversary obtains half of the password, only the second half needs to be cracked.

Other notes on methods and accounts

Of course, “ideal” break glass methods typically require cooperation and configuration from the vendor. For example, with regard to break glass accounts, most vendors provide administration authorization that is universal administration, not limiting the account authorizations to “only account creation and management”. With this in mind, be conscientious in creating break glass methods that can be implemented on the systems that are being managed.

E. Concluding remarks

Dealing with adverse situations is the foundation of business continuity planning. The situation of losing access to a system or server is no different than any other adversity. Break glass access methods are part of the recipe of a comprehensive recovery plan.

I hope this article has been helpful! If you have any recommendations please drop me a line.