In general, InfoSec organizations should be doing much of the planning that should have been done previously.  Now is the time to review patch management policies, discuss staffing issues with operations teams, and make sure that disaster recovery and business continuity plans are ready to deal with any crisis that may (or may not) happen in the future.

The Denial of Service Issue Explained

Robert E. Lee and Jack Louis of Outpost24, a Swedish security firm, recently disclosed fundamental problems with the TCP protocol that could lead to a Denial of Service.  They’re currently working with many different vendors to create patches and/or workaround solutions, but none are currently available.  As a result, the issue has only been partially disclosed, and the seriousness of the issue hasn’t even begun to be evaluated yet.

Denial of Service issues with Internet protocols are nothing new, and quite a few have been discovered over the years.  The difference with the Outpost24 discovery is the few resources that are required by the attacking host to successfully accomplish a Denial of Service.  According to Robert E. Lee, “from under forty packets per second, we could probably take off most TCP services that we interact with.”  He goes on to note that there are several different attack scenarios, several of which will result in the attacked host being taken offline until it is rebooted.

The important part of the discovery is that all hosts that have been tested by Outpost24 have been found vulnerable, including Windows, Linux, and BSD servers, as well as routers, firewalls, and other network components.

The best source for information about the issue is the two podcasts that Robert E. Lee has recently done.  CurbRisk has created and posted transcripts of the original podcast with a Dutch blog, as well as a followup with SC Magazine.

What this Means to Enterprise IT

Patching is certainly nothing new to enterprises.  Any enterprise that uses computers has had to deal with Microsoft’s “Patch Tuesdays”, and other patches being released for other operating systems.  The DNS vulnerability announced by Dan Kaminsky earlier this year was somewhat unusual, as it affected all DNS servers on multiple operating systems.  This issue is very unique, as it affects all network-connected devices running any service on any operating system.  IT organizations have never faced the challenge of patching all servers and networking devices simultaneously.  As a result, it’s unlikely that many enterprises have plans in place to accomplish such a significant, previously unheard-of scenario.

Steps IT Organizations Should be Taking Now

Although the impact of this issue has yet to be fully understood, and it’s unknown whether vendors will be providing patches, it’s important to be prepared for anything that might happen as a result of this vulnerability.

Asset Prioritization

Update your list of assets, and ensure that you have full awareness of any critical part of the infrastructure.  Be sure to include critical servers, routers, firewalls, IDS and IPS sensors, and any other part of the infrastructure that is important to the organization.

Create a Patching Plan

Work with your operations team to create a plan to quickly patch all of the critical assets in your organization.  Identifying each asset in the order of its importance will make it easier to get the patches out to the most important systems first.  In most cases, your network devices will be the most critical items.  Also consider whether it’s appropriate to start patching your Internet-facing assets first, if your organization relies heavily on email or websites to communicate with clients.

Check your Business Continuity Plans

Make sure they account for the possibility of a sustained outage of the entire corporate network, or even an outage of the Internet.  While it’s unlikely this vulnerability will take down the entire Internet, it’s a possibility that should be considered, and planned for.  Even if this issue isn’t the “internet killer” that some have hyped it to be, there’s always the possibility of another vulnerability down the road with that devastating effect.

Verify Redundancy

If your organization does not have multiple carriers for Internet access, now might be the time to add some redundancy to your connectivity.  Multi-homed Internet access is a best practice for any organization that relies upon its Internet connectivity for business.  If someone manages to use this vulnerability to take down the routers at your primary carrier, you’ll want to have another for backup purposes.

Consider Using a Content Delivery Network

Content Delivery Networks like Akamai have the ability to cache your website content in many locations throughout the world.  If your business relies on its websites to be available, a CDN might prove to be a valuable option to keep your sites available during a potential attack.  It would be best to protect your source server, as the CDN will need to contact it to build and update its cache.

Plan to get Operations Staff to Colocation Facilities

If you rely on “remote hands” staff at a colocation facility, understand that they’re likely to be completely overwhelmed by the volume of requests if a massive, critical multi-vendor patch is released.  At some facilities, they’re likely to be tied up managing the patches for the facility itself, and may not have time to work on client equipment.  If your colocation facility isn’t in close proximity to your operations staff, start planning how you’ll get them onsite to handle patches.

Start Communicating Within IT

Begin the process of speaking to IT executives, to let them know that you’re on top of the situation.  If and when this story starts making the rounds of the mainstream media, it’s always best to be sure that executives know that you’re already aware of the potential for an issue, and planning is in process.

It’s also time to start alerting the operations teams that something big may be coming, so they can consider staffing needs, and begin their own planning.

Communicate to your Users

Your users don’t need to know the specifics of this issue, or even that there is a potential issue.  However, as part of your regular security awareness communications to users, stress the need for them to be very aware of anything they download from the Internet.  If this vulnerability starts making its way into botnet or malware code, you want to ensure that your users aren’t likely to take down your network with that great game they found while browsing the web.

Don’t Panic

There’s much to be learned about this vulnerability as time progresses.  It’s very possible that this is an urgent, serious issue that may have a tremendous impact on enterprises.  It’s also possible that there’s an easy workaround, or the issue isn’t as serious as it seems.  Start planning now, so that you don’t need to panic later.

The full text of the SC Magazine interview follows.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Dan Kaplan: Hello everyone, and welcome to the SC Magazine podcast for the week of October 6th. I am your host Dan Kaplan, Senior Reporter at SC Magazine. And today we are pleased to be joined by Robert E. Lee, who is the Chief Security Officer of Outpost24, and that’s a security firm based in Sweden. And Robert and another researcher there have made a potentially very big vulnerability discovery that could affect any device that accepts TCP connections.

So, something potentially pretty devastating there, and Robert, we are pleased for you to join us. So, thanks for being with us.

Robert Lee: It’s my pleasure. I’m glad to be here today.

Dan Kaplan: For our listeners, you know, talk a bit about exactly what TCP is and how a vulnerability like this might affect, you know, them — the average user.

Robert Lee: Well, TCP is one of the many protocols that your computers that are on the internet are using to communicate with one another.

And what this vulnerability is really getting at is when you try to make a service available, let’s say that you’re trying to run a web server, you’re trying to run a mail server — something that you expect a lot of anonymous third-parties to try to connect to. What we’re seeing with this vulnerability is just the very act of making that service available, you could potentially have people that want to make that service unavailable tie up all the resources to make it so that the legitimate people that you would normally want to serve web pages or communicate email with can no longer talk to your service.

Dan Kaplan: And tell me what that can mean. That means that a device could potentially be taken down? Does it mean that something as big as a data center can come down?

Robert Lee: Well, OK, so there’s been quite a bit of overhype on this issue. In the most simple form, what we’re talking about is simply saying this service that you tried to make available to everyone on the internet is now no longer available to anyone.

There are some additional attack types that we’ve been working on that can more devastatingly affect the system. So, there are ways now that we’ve established through communication we can trick the kernel into getting into different timing states where it’s taking up kernel memory, and system memory in some cases- depending on the application that we’re actually interacting with, to the point where the entire system could need a reboot before it could start functioning again, before it actually can communicate on the network again. And, in some cases, we can actually get a device, depending on the actual operating system and the application that we’re interacting with, we can actually get a device to reboot itself.

So, it does range in severity, all the way from the most basic level of this service that you were trying to make available is no longer available, all the way up to a system that needs to be rebooted.

Dan Kaplan: Now, is this something, you know, theoretically if this vulnerability were to be exploited, is it something that is targeted meaning, you know, it would be one person trying to establish a TCP connection that, you know, can’t be closed - or is this something that could affect huge numbers of people all at the same time?

Robert Lee: I would guess that it would be more targeted. We don’t really know that for sure because we haven’t really tried to put on our attacker goggles for this issue yet. We’re still in a very scientific method trying to say, OK, we sent this very small stimulus and we’re measuring this response from the other side.

And so, that’s the extent of our research right now. We have not taken this into a large-scale, weaponized, exploit proof of concept form, but in our labs for every device that we set up, no matter what type of device it’s been so far, if it’s running a TCP service on the smallest scale, on the more- in a universal way- any service that’s available we can easily take offline.

Now, that in and of itself doesn’t seem like that big of a claim because as Fyodor and some other people have been posting online, that’s been true for a while in lots of different ways. In fact, there are even ways where with a bash script and a dummy kit you can if you know a computationally expensive page to ask for on the remote side, you could script that to where you’re constantly barraging it and hitting CPU limits on the other side if you know exactly what page you’re asking for. But those are all very simplistic ways of thinking about the attacks that Jack has actually discovered.

What we’re doing is very, very pinpointed for the stacks that we’re interacting with. So, I think to think about it in those terms. We’re not getting any new level of unavailable other than, you know, again, the service that you’re trying to interact with isn’t available and it stays unavailable until the system is rebooted. That’s the biggest difference between the more simple attacks and the advanced attacks that we’ve been talking about.

Dan Kaplan: Now, what you guys discovered, these advanced attacks, this was something that you almost stumbled across by accident.

Robert Lee: Well, sort of. Jack is the programmer that’s behind Unicornscan program, which is…

Dan Kaplan (interrupting): Jack — I don’t know if I mentioned — Jack Louis is a researcher at Outpost24 who worked with you on this?

I heard from Robert E. Lee via email earlier today, and he has indicated that he’d like to make some corrections to the comments that are shown below. I’ll be speaking with him soon, and will update this page with his comments. I’m in the process of transcribing another podcast which will be posted soon, and will also have an article here on CurbRisk with some suggestions for what enterprises should (and should not) be doing in response to the news reports of these vulnerabilities. Be sure to subscribe to the RSS feed, or subscribe via email to stay up to date. Links to both are on the top right side of this page.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Interviewer: Robert Lee and Jack Louis, you’re both of Outpost24. And using basically, your Unicorn scanner you did some amazing discovery. First off, what is Unicorn and what are we talking about?

Outpost24: Okay, well Unicornscan was a, an attempt to make a userland TCP stack. We were originally using it to collect information from large networks that we were being paid to do penetration testing against. I’ll let Jack briefly explain how that works. I, I guess we were doing the tests and we just couldn’t get the port scanning done in time. And so we decided to move the TCP stack into the program, so we could make it distributed. And one of the problems you run into when you make a distributed stack is that the state of the stack has to be, I guess you could say it has to be contained in some sort of way, to where each machine doesn’t have to track all the TCP connections. Which is why we kinda had to take a strange approach by using the reverse SYN cookies or whatever you want so we could do the three-way handshake part without necessarily tracking the connection each step of the way. And so that’s basically why or how we started noticing some funny things while scanning various networks. One other thing, too, that was important was Unicornscan was a really fast port scanner too. Which, of course, it would lead you to sometimes accidentally hit, like, network windows where the connectivity was bad, and so you would experience packet loss. And we didn’t really have much code for determining if packet loss was happening. And, so we, in the beginning at least, we would hit periods where we would just experience packet loss.

Interviewer: Wait, so packet loss basically then you mean that the system runs into problems?

Outpost24: Yes, and so we were noticing that certain stacks, and I think we were mainly seeing it with routers, would randomly end up in strange states on the other side when there was packet loss. And I guess that’s basically how we discovered it. Because a lot of these stacks, we would hit certain states inside of their TCP where they basically would just never give up trying to talk back to us. It would retransmit certain packets over and over and over again until the device was actually rebooted. You know, we didn’t right away go into, I mean we thought it was strange, you know, so we took note of it. But we didn’t really look that much into it at first. Until, I think a little bit later on when we were just curious about what, what we were actually doing to get these routers to never give up on trying to retransmit to us. I guess, kind of as a joke the first couple things that I tried, we made a program called Sockstress, Which basically was very similar to Unicornscan, but instead of establishing or trying to do the right thing while establishing the TCP connection, it intentionally did some very evil things during the negotiation of the handshake. And that’s basically what we came up with our first version of Sockstress that had two attacks in it. And I guess we were both surprised about how effective it was at first. I don’t think either of us really had any idea that it would work so well. Looking back on it, Jack before he wrote the first attack, said "Wouldn’t it be funny if this worked?", just said it kind of jokingly and then he actually did it and it, yeah it was very effective. Yeah, it worked really well so. I guess…

Interviewer: Okay, now you need to explain what worked really well, because I understand what you do is something with TCP/IP handshake. And specifically the part that has been added because there is something called like a denial of service SYN flood attack.

Outpost24: Yeah, this is definitely not a SYN flood

Interviewer: No, no, no, no. But you, you are messing with the part, what I understood, that has been added because there is something like a SYN flood attack.

Outpost24: Yeah, let’s, let’s take it back a step then. So back in 1985, there was a research paper written by Morris that talked about how you could spoof a connection attempt from a trusted host. And, you know, most people remembered the paper for that, that side of it. But one of the interesting elements of the spoof detective he describes, is finding a working service on the host that you want to spoof, and SYN flooding it to make it so that when you do send the SYN packet spoofed from its IP address and the SYN-ACK comes back to it, it doesn’t reset the connection for you. So that was back in ‘85 that we knew about SYN-flooding which basically is the idea, or premise, that a system with a listening TCP service has only a finite amount of resources. Let’s say hypothetically, that the system can only accept 1000 concurrent connections from different clients that are out there. The way a SYN attack works, a SYN flooding attack, is the attacker simply sends enough SYN packets to take up all those slots, or all those resources on the server side to make it so that no legitimate connection attempt has a chance to, to actually complete the connection. So in 1996, a number of different programs and articles were written about the SYN-flood attacks, this had become very popular. At the tail end of ‘96, researcher Bernstein came up with the concept of SYN cookies. And what SYN cookies basically does is that it off-loads, it basically makes it so that instead of taking up all those resources at the time that the initial synchronous packet is received, it can take all the session details through the meta data for that session connection attempt and put that into a hash table and take a look at the entry at the hash table and use that as a sequence number for its response to the synchronization attempt. And that way, the attacker now has to acknowledge this special token, this sequence number, in a valid format in order to complete the 3-way handshake. And so this basically makes it so two things happen. Number one, the attacker now has to actually receive the response packet to his SYN and he also can no longer be doing this from a spoofed location because the response packets have to be at a network that’s accessible to him. So now we are caught up to 1996. What happened shortly after that, from our research as soon as 1999, maybe even sooner by other researchers who didn’t really publish their works, but some researchers figured out that you could, in a client program, implement your own idea of client site SYN cookies. Where basically you encode something that’s your initial sequence number so that when the server responds to you, you can quickly tell if it’s a response to something that you generated. Ok, so I’ll go let Jack take over again.

Julie Murphy Casserly, a Certified Financial Planner and author of "The Emotion Behind Money" started doing business with LPL last year when her broker-dealer, Associated Securities, was acquired by LPL.  She’s concerned that the SEC went too far in censuring LPL, noting, "we need to help people in the public and protect them, not just protect them.  If regulation gets so tight and I’m not able to do my job which my clients have asked me to do, then what are we all doing in this industry?  Truly, this raises a concern that regulators will not take into account what the consumer is asking from us as advisors."  She’s pleased with LPL’s technology, noting, "LPL is one of the last broker-dealers that has the technology sophistication for me to do my job easily and allow me to properly service my clients."

Others felt that the SEC didn’t go far enough in reprimanding LPL.  Aaron Gordon, vice president of Schwartz Media Strategies, a Miami, Florida public relations and marketing firm, is a customer of LPL Financial.  "I think the censure was appropriate. However, the fine seems low to me. Considering how much money was at risk, $275K seems arbitrary, if not too low. Perhaps another approach would have been to fine LPL a percentage of the assets exposed to risk by their negligence."

Aaron told me that he has been a victim of debit card theft in the past, and that experience increased his concerns about the security of his personal information.  "Within 24-hours, $7,000 had been debited from my checking account. Thankfully, Bank of America refunded my money, but it was a hassle. Since then, it’s always on my mind when I swipe my card or make online transactions.  The safety of my information is of utmost importance. I think data security should be a given in this day and age."  He also feels that financial services companies should be held to a higher standard: "If they’re going to take on the risks associated with keeping highly-sensitive information, then it’s their obligation to protect that information."

Despite his concerns, Aaron is keeping his money with the independent financial adviser that uses LPL for his investments.  "For me, it’s about my relationship with my adviser. I trust that the government will take care of the problem and that LPL will ensure it doesn’t happen again."

Tina Trombley, an IT Resource Project Manager from Ohio, is also keeping her money with LPL.  She told me that she became aware of the SEC’s censure and fine of LPL Financial when news of the current global financial crisis caused her to do some research in Google regarding LPL’s financial stability.  The hack of LPL’s BranchNet system hasn’t caused her to change her relationship with LPL, at least not yet.  As she puts it, "I am not removing my money from my accounts with LPL, but I am watching them very closely and exploring other investment options for my long-term retirement investment accounts."  She also noted that she’s not worried about her data being exposed in a security breach.  "I am not overly concerned about my personal information and account as I do periodic credit checks to be sure my identity is safe."

In short, LPL Financial performed a security audit on one of its Internet-facing trading systems.  The auditor’s report included details of a number of significant shortcomings in the system.  BranchNet had no requirements for password length or complexity, and the plain-text passwords of all of the system’s users could be viewed by approximately 300 people in the IT department.  A year after LPL was aware of the vulnerabilities, they hadn’t taken any action to correct them, and the system was inappropriately accessed to make unauthorized trades in customer accounts.  In the SEC’s action against LPL, they called the company in “reckless disregard” of regulatory requirements for the protection of customer data.

I spent some time reviewing the SEC’s press releases for the past 18 months, and found no other actions taken against any company as a result of an information security problem.  The vast majority of censures and fines are related to fraud.

So, would it be better to be blissfully unaware of your security issues, or be fully aware, and just refuse to act?  Most laws in the US are written so that active or constructive knowledge (or “knew or reasonably should have known”) can be used to make a legal case against someone.  In other words, you can’t claim innocence if you should have been aware there was a problem.  However, in the minds of the people in a regulatory agency or on a jury, is it worse if someone knows and does nothing, or if they could have known but didn’t?  It is, after all, people who decide the punishment based on the facts of the case.

I don’t mean to suggest that ignoring the need to perform a security audit is an acceptable substitute for fixing vulnerabilities.  Nor do I mean to suggest that LPL would have been better off without a security audit.  However, if a company knows there’s a vulnerability, the company must act to correct it, or it risks significant liability if and when the vulnerability is exploited.  The reverse of that statement may also be true in some (albeit very few) cases.  If you know that nothing will be done to fix the vulnerabilities in a system, you might be better off not knowing about them.

As Evan notes, this is not the kind of theft that we see reported often. The thief entered the company’s offices during regular business hours, entered the office containing the laptop, cut the security cable, and stole the laptop. Nothing else was reported missing. The laptop contained the name, address, and checking account number of a “high percentage” of the bank’s mortgage clients.

I hate to jump to conclusions, but this is either the most interesting coincidence I’ve ever heard of, or someone with inside information about where this data was stored has some motive to steal this particular laptop. A spokesperson for bank has noted that since there were no Social Insurance Numbers as part of the data, the impact is “minimal”. I think the hoops the thief jumped through to steal this particular laptop makes it likely that he or she had some intention to use the data on the machine, which makes the likely impact of this theft much greater than the average laptop stolen in a crime of opportunity.

Photo by Oskay

Google has a truly admirable record for releasing products that have no known vulnerabilities, even after being released for several years. Secunia shows no reported vulnerabilities in Google Earth, no vulnerabilities in Picasa, 1 vulnerability in Google Toolbar (rated “Less critical”), and no vulnerabilities in Google Talk. Google should be congratulated for their security ability and obvious commitment to secure desktop software development. They don’t, however, have much experience in developing and delivering enterprise applications.

Among Chrome’s competitors, I’m concerned about Internet Explorer’s many vulnerabilities over the years (Secunia lists 31 advisories and 62 vulnerabilities), but I appreciate the granular control available through group policy. Firefox is my browser of choice, although I’m not impressed by Firefox’s also less than admirable vulnerability record. Both of those browsers have been well-tested, and many of the vulnerabilities have been discovered and fixed.

Google Chrome enters the fray with a new and untested product. I sincerely hope that Google’s excellent vulnerability track record with their other products is reflected in their work on Chrome. Unfortunately, since it’s so new, we simply don’t know whether Chrome has any significant vulnerabilities waiting to be discovered. I won’t trust the security of Chrome until it has been well-proven. A few vulnerabilities in Chrome have been found, announced, and patched, and it’s likely that others are out there waiting to be discovered.

There’s plenty to learn about using Chrome in the enterprise, including the speed with which Google releases patches to the vulnerabilities that will inevitably be discovered. How those patches are managed and distributed, and how Chrome is managed by IT organizations is also something that remains to be seen.

Chrome is a very early test version of a browser that Google will release for general availability at some point in the future. It’s open for download to anyone who would like to test it. Despite Google’s overuse of the word “beta” for products that are really in general release (such as GMail), Chrome truly is a beta product. Additionally, it’s a beta product with a significant and important impact on the security of machines used to test it. Browsers continue to be the leading point of undesired entry into workstations. They’re security-critical software, and the importance of secure browsers in enterprises cannot be understated.

Test software does not belong on enterprise machines used for business purposes. Regardless of how much users may wish to try out the latest and greatest technology, enterprise machines need to be properly managed, and kept stable and secure.

All of that said, the decision should be a simple risk/benefit evaluation. Answer the question, “what is the business benefit of Chrome?”, and compare the answer against the potential risks. I haven’t yet heard of any business benefits from the enterprise use of Google Chrome compared to current browsers, which makes the risk/benefit calculation quite simple: Google Chrome simply isn’t ready for enterprise use.

If your enterprise hasn’t yet stopped users from downloading and installing Chrome, now is the time to do so. It’s much easier to stop the use of a product before it’s widely used than to suffer the outcry when it’s taken away.

When Chrome is officially released, or when it has been used long enough to prove itself, reevaluate whether it’s still appropriate to block it in your enterprise, and adjust appropriately. I, for one, hope that Google Chrome brings some more competition to the browser market, and produces some valuable innovations for enterprises. Until that happens, though, it’s not ready for enterprise use.

Update (September 27): Zero Day at ZDNet reports on a newly discovered DoS vulnerability in Chrome. Yet another reminder that we don’t yet know what vulnerabilities await discovery.

After receiving the internal audit, and after months of consideration regarding the cost/benefit of the security remediation, LPL’s Executive Risk Committee doesn’t appear to have ever reached a decision on whether to fix the problems in BranchNet.  While they were deliberating, accounts of LPL’s customers were accessed to make unauthorized trades over an 8-month period beginning in July 2007.

In their cease-and-desist order, the SEC says that as a result of LPL Financial’s failure to take immediate corrective action, the company was “in reckless disregard of the regulatory requirements.”  LPL accepted the censure and fine without admitting or denying the SEC’s findings.

Interestingly, the SEC notes that LPL detected the unauthorized trading attempts, and was able to block most of them.  However, if they were aware of the intrusion, how did they allow it to continue for 8 months?

The password policies of BranchNet read like a case study in what should be avoided with passwords:

Regarding password complexity, LPL’s internal auditors identified the following weaknesses concerning the BranchNet application: (1) RR passwords did not meet industry standards for so-called “strong” passwords, because, among other things, the passwords had no requirements on length or alphanumeric/special character combinations; (2) passwords were not set to expire after a certain period of time; (3) users could not change their own passwords; and (4) there was no automatic lockout feature related to unsuccessful login attempts. Additionally, over 300 LPL information technology employees had access to a list of BranchNet passwords, and a number of former employees likely had access to such a list before leaving the firm.

With respect to BranchNet session inactivity, LPL’s internal auditors observed that the automatic session timeout was set at eight hours, which LPL’s internal auditors believed was significantly longer than the timeout periods used by other financial services firms for similar applications.

This is for an Internet-facing application that manages accounts and trades for customers of a company with more than 1 million customers and 282 billion dollars in assets under management.  Based on the description above, it would appear that a user could set a password equal to “a”, and the system would accept it.

As I researched a bit more, I found this interesting tidbit from LPL’s customer loss letter at BreachBlog:

In March 2008, LPL hired Marc Loewenthal as SVP - Chief Security/Privacy Officer, a newly created position at LPL.

Chief Security Officer is a newly created position at LPL Financial?  I’d ask who runs their Information Security department, but I’d be afraid that I’d hear they don’t have one.

LPL also began reporting breaches related to laptop loss incidents immediately after the accounts on BranchNet were compromised.  I’d venture to guess that they were unaware that data losses needed to be reported until they started working on this major incident.  PogoWasRight.org shows 5 laptop loss incidents reported in the last 4 months.

Although the SEC’s action only covered the breach that occurred between July 2007 and February 2008, PogoWasRight reports that another password was compromised in May 2008 resulting in the loss of data of 185 individuals.

It’s pretty shocking to me that a financial company with $2.7 billion in annual revenue can have security issues anywhere close to the problems at LPL Financial.  I wish Mr. Loewenthal great success as the new CSO, and I hope he knows what he’s in for.

The SEC’s order is here (PDF): http://www.sec.gov/litigation/admin/2008/34-58515.pdf

(Post updated on September 23 to note the “reckless disregard” comment from the SEC)

Though domain names are (obviously) crucial to a company’s Internet presence, many companies are lax in their procedures to ensure their domains are managed properly, and have allowed their domains to expire, disrupting service to their customers.

In July, RSA forgot to renew securesuite.co.uk, the domain name it uses to manage secure credit card authorizations for merchants in Europe.  A few months earlier, Google forgot to renew grandcentral.com, the domain that hosts its GrandCentral voicemail service.

The list of organizations that have accidentally allowed domains to expire includes Microsoft, The Government of Greece, The Washington Post, Electronic Privacy Information Center (EPIC), and the Eureka, California Police Department, among many others.

A few relatively easy steps can go a long way toward reducing risks that can have a major impact on a corporate Internet presence:

Register or renew your domains for as many years as possible

Generally, domains can be registered or renewed for up 10 years, and Network Solutions will even allow registrants to pre-pay for 20 or 100 year terms.  The easiest way to ensure your domain doesn’t expire is to make sure that it isn’t scheduled to expire for a long time.  The expiration date should never be less than 1 year away.

Who isn’t doing this among the Fortune 100?  At the time I’m writing this, Wells Fargo’s wf.com is scheduled to expire within 5 months, Humana.com expires in about 2 months, two of HCA Healthcare’s important domains expire in just over a month, and Tyson Foods’ corporate domain expires in only 3 weeks.

Register all of your domains with one registrar

Corporations often own hundreds or thousands of domain names, and it can be very difficult to manage domains spread among many registrars.  By keeping all of your registrations in one place, it’s much easier to ensure your contact information is kept up-to-date with the registrar.

Seek out a registrar focused on serving businesses

The registrar business is often a mostly-automated endeavor, and most registrants need only basic domain-related services.  As a result, the biggest focus among registrars is to reduce prices as much as possible to attract customers.  Businesses have different needs than consumers, and a human account representative should ensure that domains aren’t allowed to expire accidentally.  Seek out a registrar who can provide personal service.

List a generic role as the domain registrant

When registering a domain, the registrant is often asked to name an administrative contact, a technical contact, and a billing contact.  If an individual person is named in these roles, that person is the only one who has complete control over the domain, and problems can arise if that person ever leaves the company.  Potentially worse, that ultra-admin’s name and email address are made publicly available through whois, which could make social engineering attacks much easier.

The best practice would be to name a generic role, such as "Domain Admin", with an email account that forwards to multiple people or to a help-desk queue.

Unfortunately, 23% of Fortune 100 companies list an individual’s name or email address in their publicly available whois data.  I’d be interested to know how many of those individuals no longer work for the company whose domains they control.

Use periodic audits to check the status of domains

If you already periodically audit the organization responsible for managing your corporate domains, add an audit step to verify the domains are not near their expiration date, are registered properly, and the passwords to the domain registrar account have been recently changed, and are kept secure.

If you don’t currently audit the organization responsible for your company’s domains, perhaps it’s time to consider adding them to the audit cycle.

As a result, people within the enterprise often fear the security organization, and avoid it whenever possible.  In a previous life, I was an application development manager, and know the feeling first-hand.  People representing the business and the IT staff often need help managing risks and "doing the right thing" to secure their systems and business processes.  Given the resources they need, most people will make the effort to act responsibly, and seek out help to reduce risks in their project, technology, or business process. Without the right help available, most people will focus on what they do best, and ignore otherwise important but confusing risk-management activities.

Security organizations exist to serve the needs of the business.  We are service organizations, and we’ll only succeed when we provide valuable services to the business (and their representatives in IT).

There are some steps security organizations can take to act more like the service organizations they are.

Put up roadblocks only when needed

It’s easier to get people to act when there’s something standing in the way of success.  People pay attention when they’re about to miss a deadline or blow their budget, and stopping a project dead in its tracks is a sure-fire way to force a change in course.  It’s also a good way to lose the respect of the project manager, and make sure they avoid you in the future.

There are often plenty of good reasons why the risks of proceeding with a project outweigh the business need to complete the project on time, within budget, or at all.  Understand these reasons, and articulate them to the business representative before you delay a project. If you can’t articulate the reasons to delay a project, re-evaluate your risk assessment.

Use responsiveness to succeed where others fail

IT organizations are sometimes filled with people who aren’t responsive or helpful when people reach out for help or information, and security organizations often make the same mistake.  Pretend for a moment that you’re a consultant, and the person who is reaching out for help is a paying client.  Help them as if your job and the success of your organization depend on providing quality service.  Repeat this pretend exercise enough times to understand that the business really does pay for the security budget, and they expect to get some value in return.

There are reasons for every policy — understand them

Policies exist for a reason.  Every member of your organization should understand the policies, and the reason why they’re there.  Someone in your organization should understand the policies well enough to understand the purpose behind each of the policy statements, and be able to explain it effectively to others.  If you can’t find a good reason for a policy statement, it doesn’t belong in your policy.

Don’t give up!

It’s not easy to change the groupthink sometimes prevalent in security organizations.  It takes some work, takes some practice, and there will be some failures.  Keep at it, and don’t give up.  In the end, others will stop running away from the security organization, and the process of reducing risks will be much easier.

CNN spam messages are still arriving at a quick pace, and I’ve received a number of them today, most recently about 10 minutes ago. They’ve changed again, and the title of the supposed article in the alert now has a link to the malware. Previously, the article titles linked to a legitimate CNN article describing a threat on the Olympics in Beijing, and only the link titled "FULL STORY" linked to the malware. (See previous post, "CNN Spam Contains Mostly Legitimate Links")

I’m also starting to see links to html files with different names, as others have reported, including cnncurrent.html, cnnhottopics.html, and cnnheadlines.html.

It appears that CNN is paying attention, as someone visiting from CNN’s IP space read my original blog article just a few hours after it was posted. They found my blog by searching Google for "cnnplus.html", one of the filenames used on compromised servers to spread the malware.

A sampling of the some of the articles linked in the spam I’ve received is below. Note that some of them are real events in the news, while others are completely false:

• Bank of America files for bankruptcy, to be taken over by Feds
• Woman Attacked by Beau’s Pitbull
• Top Italian car designer killed. See the video
• Bill Gates: The end of the Windows era is coming
• Two arrested after using barbecue pit as a weapon
• Coup demonstrations in Mauritania

Some other sites with great information about this variety of spam:

Michael Roberts at Vivitek has posted a very detailed list of compromised domains hosting the malware HTML pages.

Spyware Techie has some information about Blogger comments used to spread the spam, as well as a very detailed CNN Spam trojan removal guide for those who have been infected.

Update (11-August): The spam seems to have changed again, as I’m now seeing links in the emails to a file named cnnvid.html.

Update (15-August): I haven’t seen any CNN spam in quite a while, but I’m now seeing lots of MSNBC spam.  Perhaps Fox News is next?

CNN Spam

Also interestingly, the images in the spam are hosted at http://i.a.cnn.net, which tells me that CNN has the ability to include a spam warning, instead of their logo on the spoofed email. At the time I’m writing this, they have not chosen to do so. Although I’m sure the logo is used elsewhere, they should be working on using this method to provide a warning, given the massive numbers of spam being sent right now.

Some other interesting items:

• I’ve received two of these messages. One of my addresses was culled from a whois record, the other is a random string of characters sent to a domain that has “catchall” enabled.
• Both spam messages link to this CNN story about a threat at the Olympics, and neither of the story titles in the spam message are actually on CNN.com (as far as I could find).
• One linked story is titled “Heath Ledger’s body exhumed”
• The other story is titled “Depression and sadness will kill you, so lighten up here”
• Both of the spam messages I received linked me to the same filename at the root of two different likely compromised domains: cnnplus.html

Terry Zink also has some examples of the original CNN “top 10″ spam on his site.

Although it’s not an information security control, the perfect example exists in my office. In the 12 years I’ve worked for this company, the ladies’ restroom has always had a combination lock on the door. The three digit code has always been the same, and was always shared with all employees. I’ve never been in the ladies’ room, but have always imagined the lock was there to protect a secret connected room with luxurious furnishings, a fully stocked bar, and butler-passed hors d’oeuvre.

The locked ladies’ restroom wasn’t usually a significant issue for me, as the men’s rooms had never had a lock. However, when I was visited by female vendors or job candidates, the lock was a significant issue, especially when the meeting occurred after hours or on weekends. Usually, there was always a female employee around who would provide me the code, but it was always provided with a "why do you want to go into the ladies’ room?" facial expression.

All was fine for 12 years, until we moved to a new office building earlier this year. In their infinite wisdom, the facilities department decided the men’s rooms should now also have combination locks. The reasons for the locks have never been explained. As an occasional user of the restroom, I cannot possibly imagine there’s anything in there worth securing. Although it only takes 3 seconds to enter the combination, the annoyance level is high. On several occasions, people (including myself) have found themselves locked inside the restroom when the mechanics in the lock failed, and the door would not open.

To the surprise of nobody, people with legitimate access to the restroom have bypassed the security control using tape or paper towels. Each day, someone from the facilities staff would remove the tape, only to find it in place again the next morning. After several weeks, facilities gave up, and the door can now be pushed open without using the lock. In the process, facilities became known as "the group that locked the bathrooms." Although disabling locks in the building could certainly be an issue worthy of official reprimand, that cannot reasonably happen unless someone can demonstrate the harm that was caused. In this case, I’m at a loss to find one.

I’ve always kept this example in the back of my mind when evaluating risk, and refer to it when considering which technical security controls to recommend. If there’s not a clear security benefit to be gained and user frustration could be an issue, it’s probably wise not to implement the control.  Or, as I like to think of it, it’s best to just leave the bathroom unlocked.