Thursday, April 15, 2021

Time to Train -

 "Excuse me, sir. How do I get to Carnegie Hall?"

"Practice, Practice, Practice."

I've always said that a poorly trained sysadmin is one of the greatest threats to any organization's infrastructure. The military training module may seem archaic and cumbersome but it is effective. There is a significant amount of investment in creating an effective training program. I believe the correct technical description is "it ain't cheap".  Organizations that fail to train their technical and general user staff in basic or advanced IT security practices are doomed to suffer multiple failures. 

I'm not going to dive into pedagogy (can't help but giggle everytime I hear that word) or the merits of a good training program. Too much has been said on those topics. Instead, I'm going to present my idea of a training roadmap here:

 Here we have 3 main training tracks:

  • Technical track - the target audiences are system administrators, developers, IT Security analysts/architects. These training programs are designed to enhance your staff's technical knowledge.
  • Awareness track - the target audiences are your general staff, management. These training programs are designed to make your workforce aware of the laws, regulations, best practices for handling your organization's sensitive data. In addition, these programs show your staff the different types of physical and cyber attacks they may see and how to respond to these threats.
  • User (How-to) track - this training program teaches your staff how to use the day to day tools of your business. It covers things like how to:
    • use Microsoft Office, Adobe Acrobat tools
    • use graphical design tools
    • use collaboration tools
    • use in-house tools
    • use external software or hardware products.
There needs to be a blend of externally developed training materials (SANS Secure the Human, Skillsetsonline, LinkedIn Learning, etc.) and "local" training for in-house applications.

Take a look at the above roadmap and I would like to hear your suggestions on how to improve or implement the roadmap.

Saturday, January 23, 2021

Resilience Is the Key to a Successful Defensive Strategy

The main mission of any CISO is not to prevent breaches of their infrastructure, rather, it's to safeguard you organizations' sensitive data and identity. I've said many time in the past that there are no device breach notifications but there are plenty of data breach notification laws. There are many ways to protect data and identity like encryption, monitoring outbound traffic, increasing user awareness, multi-factor authentication. These are important things but they are a means to achieve a goal. Resilience is the key to a sound defensive strategy. Here are some thoughts.

  1. We play defense not offense. 95% of companies hire cybersecurity people to defend their company from cyberattacks. They don't hire them to attack other sites. That's what the remaining 5% do. However, to play good defense, one must know how to play good offense. In other words, a Blue Team should have strong Red Team skills.
  2. One must accept the fact that a breach will happen regardless of whatever controls are in place. The old defensive strategy of building a "wall" to keep the bad guys out has failed. While there are many variants of the now popular Zero Trust Network philosophy, there are 2 key points that must be in place:
    1. The network is hostile.
    2. Data and identity are the new borders
  3. The key to a successful defensive strategy is resilience not prevention.  A sound resilience strategy is key to recovery.


I could give the Webster's dictionary definition of resilience but let me give you an example.

Ransomware is one of the destructive attacks that has affected a large number of organizations and people recently.  It's been around since 1989 but what made it popular was the introduction of cryptocurrency as the payment mechanism. For example, the Virginia State prescription monitoring database was hit with a ransomware attack in 2009 and the attackers demanded a $10M ransom. The state didn't pay and restored from backups. There was a disruption of service, some loss of data but the service recovered. Collecting the $10M in small bills requires a bunch of duffel bags and every LEO in the planet watching those bags to see who collects them. 

This incident convinced me that the best defense against ransomware attacks is not "prevention", rather, it is "recovery". Take the time to carefully align file permissions with need-to-access requirements of the business. This is a difficult step. It may limit ransomware damage by limiting the files the malware can access.
A good backup strategy is the best defense in this case. A system gets hit with ransomware, you wipe it, patch it, restore your data from good backups and then move on with your business. A good resilience strategy should include these steps:

  • find your sensitive data. Consolidate it into something like a data lake. 
  • Map where your sensitive data goes within your network borders as well as outside your borders. 
  • Backup this data lake by taking snapshots, doing old school incremental backups and store the backups offline in a read-only mode. For example, NetApp devices allow the creation of a read-only snapshot.
  • Test your recovery processes frequently.

The old RFC 1244 "Site Security Handbook" describes two defensive strategies: "Protect and Proceed" and "Pursue and Prosecute".  It set the following conditions for each of these approaches:
Protect and Proceed
      1. If assets are not well protected.
      2. If continued penetration could result in great
         financial risk.
      3. If the possibility or willingness to prosecute
         is not present.
      4. If user base is unknown.
      5. If users are unsophisticated and their work is
      6. If the site is vulnerable to lawsuits from users, e.g.,
         if their resources are undermined.
   Pursue and Prosecute
      1. If assets and systems are well protected.
      2. If good backups are available.
      3. If the risk to the assets is outweighed by the
         disruption caused by the present and possibly future
      4. If this is a concentrated attack occurring with great
         frequency and intensity.
      5. If the site has a natural attraction to intruders, and
         consequently regularly attracts intruders.
      6. If the site is willing to incur the financial (or other)
         risk to assets by allowing the penetrator continue.
      7. If intruder access can be controlled.
      8. If the monitoring tools are sufficiently well-developed
         to make the pursuit worthwhile.
      9. If the support staff is sufficiently clever and knowledgable
         about the operating system, related utilities, and systems
         to make the pursuit worthwhile.
      10. If there is willingness on the part of management to

                                             Figure 1. Protect and Proceed vs Pursue and Prosecute

My ransomware scenario's recovery process is an implementation of requirement listed in RFC 1244 - " "Attempts will be made to actively interfere with the intruder's processes, prevent further access and begin immediate damage assessment and recovery." 

This is an example of resilience. Andy Greenberg's book "Sandworm" has a chapter dedicated to resilience. Dan Geer's essay "A Rubicon" is another example of the importance of resilience. Creating an "parallel" network universe addresses interdependency issues and allows for a quick recovery.

We should certainly spend funds on detection tools but the bulk of present-day defenses should be focused on how we recover from an attack. Resilience processes such as backups,  monitoring and disrupting outbound traffic to questionable sites are examples of a good resilience strategy. 

You're going to get breached at some point in time. How fast you recover can limit the damage done to your business processes. 


Monday, August 31, 2020

RDP Security Tip and other Infographics

 Thanks to Thomas Roccia for this great resource he created. It's at I think you'll find these graphs to be particularly useful in any presentation you do. 

We've been asked a lot about Remote Desktop security given the WorkFromHome (WFH) situation we're in during the pandemic. It is a serious problem and here's a great infographic from Thomas' site. 

Saturday, August 8, 2020

Academic Freedom and IT Security - They Do Work Well Together

 I was a member of a panel on Cyber Hygiene that was sponsored by the SANS Institute today. My good buddies, Tony Sager and Russell Eubanks were also on the panel. 

An attendee asked me about the challenge of balancing IT Security practices vs. the cherished Academic Freedom (AF) issue. I responded that IT has to stop being the Department of NO and go out and listen and learn how researchers do their thing. Only then should they decide on a path that supports rather than hinders their research. It's harder to take the time to meet and learn how end users actually do things  given the multitude of tasks most IT people need to perform in their normal course of duties. Understanding how and why your end users do things allows you to design and build a more efficient IT Security program and architecture. Short term pain eventually leads to long term gain. Taking the time to understand how your end users actually use your IT services will actually lessen the amount of time you have to spend outside of your normal duties in the long term. 

It was a great question and it got me thinking about the issue a little more and hence, this blog entry. I've been working in EDU IT for 45 years now and here are some musings on this balancing challenge.

I went on a motorcycle ride and got to thinking more about the question while I was riding through the mountains. It occurred to me that there should be no conflict between IT security and AF principles.  IT Security practices should enhance and protect AF. One complements the other. 

First, let's try to define "academic freedom" for the purpose of this blog. Here are some definitions that I'll use as my foundation. Academic Freedom is defined as:

1. a scholar's freedom to express ideas without risk of official interference or professional disadvantage. "we cannot protect academic freedom by denying others the right to an opposing view" (Oxford Dictionary)

2. Academic freedom means that both faculty members and students can engage in intellectual debate without fear of censorship or retaliation. (

3. Teachers are entitled to full freedom in research and in the publication of the results, subject to the adequate performance of their other academic duties. Teachers are entitled to freedom in the classroom in discussing their subject, but they should be careful not to introduce into their teaching controversial matter that has no relation to their subject. (

After reading these definitions, I tried to see what the conflict was between IT practices and Academic Freedom (AF). Frankly, I saw more opportunities for IT practices to support, secure and protect AF. All 3 of the above definitions emphasize the right of the academic community to discuss freely any topic without the fear of censorship or retaliation. Looking at this from the IT Security point of view, here are some threat scenarios to AF in the online world.  A sample threat would be attacks against the Confidentiality, Integrity and Availability (CIA) aspects of AF.

For example, let's look at censorship. DOS/DDOS attacks,  domain blocking, confiscation of servers or endpoints are examples of availability attacks. Unauthorized modification of topics/data is an example of an integrity attack. Doxing is an example of a confidentiality attack. 

There are existing IT Security practices that can mitigate the effects of these classes of attacks.  Availability threats such as DOS/DDOS attacks can be deflected. Domain blocking can be addressed. Good file permission strategies along with good backups, file integrity tools can mitigate integrity attacks. Hunting down doxxers, online "bullies" can be done using techniques such as OSINT and log analysis to protect  individuals from harassment or retaliation.

Sound IT Security practices can and should be done to further advance academic freedom. I think the supposed conflict between IT Security and AF is not the big issue everyone outside of the EDU world thinks it is. 

To the webinar attendee who asked me the question of balancing IT Security practices with Academic Freedom, let me say IT Security practices should support academic freedom by designing procedures for protecting one's right to academic freedom. It should never interfere with that core business process.

This is my short answer to this question. I'd like to hear your opinions on this matter.


Friday, August 7, 2020

Encryption, Security and Privacy, Oh My!

We’ve been hearing a lot of discussion about encryption these days. The Federal government proposes installing “backdoors” in encryption algorithms to allow law enforcement and security groups to be able to monitor communication between entities who pose a threat to “our” security.  We’ll talk more about this later but we want to emphasize this is an “age old” argument.
Clay Bennet won a Pulitzer Prize in 2002 for an editorial cartoon that expertly explains the security vs. privacy issue. Imagine a house, two people inside it and a wooden fence around the house. The house has a label that says “PRIVACY”. Workmen are removing planks from the house and using them to build the fence that has a label that says “SECURITY”.  Security vs. Privacy is like a see-saw. The more security you want, the less privacy you have. It is not a “vice versa” situation. More privacy does not necessarily mean less security. Security advocates usually say “if you’re not doing anything wrong, then you shouldn’t be worried”.  There are lots of flaws with this argument. The most common one is “who defines what is the definition of “wrong”? Does wrong mean “illegal” or dissent, for example. A common definition of privacy is the “right to be left alone”.
Encryption provides a way to hide something you send or store from unauthorized entities. It can be as basic as speaking a foreign language to someone or using something based on high order mathematics. For example, the Navajo code talkers used their language as an “encryption” method of communicating without the enemy being able to determine what was being said. As with any process, it can be used for good or evil.  You “break” this encryption technique by using someone fluent in the language being used.

In the 1990s, the Federal government proposed a method (the Clipper chip) allowing law enforcement and security groups to decrypt encrypted information. The resulting uproar was instrumental in shooting this proposal down but it showed how people didn’t understand how encryption works. The “clipper chip” was a “backdoor” way to decrypt a file or transmission. Suppose you put your tax papers in a vault to protect it from unauthorized access. You use a lock and key to gain access. A backdoor would be something like a master key for that lock that allows it to be unlocked. Common sense tells us the master key a) needs to be guarded all the time b) the person who has the master key isn’t evil and c) the person who has the regular key knows a master key exists.

So what’s the problem? Well, in the digital world, copies can be made without the owner’s knowledge. Any good hacker would try to get that “master” key and use it. It’s folly to assume a digital “master key/backdoor” would never be compromised. The 2011 RSA hack and 2013 Carbon Black attack are examples of hackers going after the “master” keys with success. While the whole purpose of encryption is to protect data at rest and in transit, there are ways to try to get the data in its original form.  Consider the following:

A -> K -> M1  ->C1-> EC -----------à DC -> C2 -> M1 -> file/display -> B

Person A uses a keyboard K to create a message M, stores it on computer C1 and encrypts it using tool EC. The encrypted message arrives at the target machine, is decrypted by tool DC running on device C2,  the data M is either stored in a file or shown on the display to person B.  The message is encrypted only from EC on C1 to DC on C2.  Attack points where the data could be copied are at K, C1 C2, M, file/display. Note these attack points do NOT need to know your encryption key. Why? The data is in the clear when it’s entered at K, stored in a file M1. If you write a program to grab the data at these points, you get the data in the clear.

This is nothing new. The first public reporting of this technique was done in 1998 when the FBI used a keystroke recorder against a mafia don’s computer. The recorder allowed them to collect information used to prosecute him. The keystroke recorder copied the data as it was entered before it was encrypted. The 2001 Magic Lantern tool and the 2009 CIPAV (Computer and Internet Protocol Address Verifier) were law enforcement tools developed to get data before it was encrypted.
These were every effective techniques and did not require a “backdoor” to an encryption algorithm.

So, let’s go back to the Privacy part of this essay. Those who advocate the “security” argument maintain there’s a need to be able to determine if criminal intent is planned. If criminals use encryption to hide intent then the government needs to be able to decrypt those messages.  There are ways to get data before it is encrypted so why the need for a backdoor? We need to remember that a message or data in a file starts as cleartext. Data capture techniques have been around for the past 20 years. Since they don’t require a backdoor to the encryption algorithm, one could assume the real target is privacy.  Why? The introduction of backdoors into any encryption algorithm destroys the algorithm as an encryption tool. The backdoor(s) will become publicly known eventually and encryption ceases to exist.

What’s truly ironic about this contention is that individuals are freely giving up lots of personal information to commercial companies.

This is a reprint of article originally posted on

Out with the Old! In with the New! Perimeter Border replaced by Data and Identity Borders - Some Thoughts

These are a few questions that I'll address in upcoming blog posts.
  • Are the industry threats your threats? Just because the magic quadrant says Threat A is the critical threat you need to address doesn't mean that it applies to your network. What metrics have you collected to determine the root cause of compromises or breaches in your org? While phishing is one of the major threats touted in cybersecurity mags, is it the root cause at your site? For example, for us, the 2 major root causes that led to breaches (big ones) that affected the entire institution were a) poor password management b) failure to apply OS and application patches in a timely manner. While we did have lots of successful phishing attacks, the consequences of those hits was limited to 1 or 2 people - the person who fell for the phish and/or immediate family. On the other hand, a sister institution found almost the opposite of our results. Phishing was a primary vector in their case. My point is that we need to take the time to evaluate the real causes of successful attacks  against our infrastructure/data/credentials and then use this information to buy/build tools/processes to address those threats. This helps us avoid wasting money on defensive tools that address 1% of successful attacks against us. 
  • The New Borders - Your Identity,  Data. I used to say (still do) that the effective security perimeter is the device and not the border. As more and more devices become "personal" and not "organizational", the border becomes your phone, tablet, laptop, server, etc. BYOD is forcing us to adapt to this new paradigm.  Mobility becomes the new data flow process. 
  • Work From Home (WFH) has drastically changed the "border".  
  • Both ends (endpoint clients, servers)  of the traditional client-server process aren't necessarily inside your traditional "border". How are you approaching the visibility issue?

Monday, April 16, 2018

Why Corporate Security Should Be Like Museums? Edus Are.

I was preparing a talk for the 2018 Educause Security Professionals Conference and was trying to think of ways to show how EDU networks are really microcosms of society. I wrote in an earlier blog that EDUs are small cities. I've said that our network security strategy is a blend of commercial and ISP requirements. It was wasn't until I ran into my friend, Christian Schreiber, who gave me the best analogy so far. As a CISO, I have to give a presentation to our Board of Visitors, our version of the corporate board of directors, every now and then. I like to use real world examples to explain our security strategy. Most board members come from the corporate world and want to know why we don't follow a corporate IT security strategy.

Well, Christian was working on a talk and he said EDUs are like a museum. At first, I thought he was going to tease us about being quaint, staid and stuffy. Rather than state the obvious :-), he pointed out the following:
  1. Museums allows all sorts of individuals into their building.
  2. Museums have high value assets and protect them with a variety of tools, technical expertise.
  3. Key assets are highlighted to make them more accessible to the public.
  4. Museums cover their interiors with a wide variety of tools.
  5. Museums focus on detecting malicious operators who may be already inside the building.
 Christian went further and give some examples of museum defense in depth:
  1. Museums have few access points but they allow free flowing access to anyone.
  2. Museums erect additional barriers around high value assets.
  3. Museums have pervasive monitoring tools: video cameras, motion detectors, laser detection systems, visitors logs.
  4. Museums have numerous active response capabilities such as: uniformed guards, on-demand barriers, fire suppression systems, moving doors.
  5. Museums have recovery systems such as insurance and tracking devices embedded in high value assets.
  6. Museums assume there are hostiles inside their buildings.
As you can see, there are Continuous Monitoring, Zero Trust Network, network forensics components embedded in the bullet items above.  They allow visitors to bring their own devices, take pictures, buy souvenirs and wander freely within public spaces. They also have restricted areas that require additional authentication and authorization.

IoT, BYOD have been forcing orgs to reconsider how their network security should be implemented. The traditional border security model will fail in the new technology model unless they adapt to a mobile user environment. I used to say the device was the border. Nowadays, I believe there are 2 new borders that need to be considered:
  1. User identity - users access their work/home assets from all over the internet. For example, EDUROAM allows members of one EDU connect to the internet using another EDU's net and the member's home institution credentials.
  2. Data - If data becomes the new border then does it matter where it's stored? If its protection schemes focus on the data element itself, then I don't believe it matters.
Given these 2 new borders then the museum defense model makes a lot of sense. This doesn't mean that you should discard the older perimeter style defenses but it does mean the combination of these layers forms the basis of a reasonable, successful  museum defense.