May 17, 2012

Part II: Protect Your Data and Your Company From an Internal or External “Hack-Attack”

James Holler, Founder, Abidance Consulting

In Part 1 of this series, we touched on some ways to make it so difficult to pull off a hack-attack, that the perpetrator will most likely want to go somewhere else and try their attack.

In this section, we’re going to address testing, maintaining and other important items that deserve your attention.

Testing

Once you have fixed all of the issues, you need to test everything to make sure it works the way it is supposed to. You must first create benchmarks in which you are testing against. Just to run a test for the sake of running a test is futile. Once the benchmark(s) have been set, you are ready to test:

  • Run port scans to ensure only required ports and services are open and/or running
  • Firewalls detect intrusions
  • Switches and routers have only active administrator accounts
  • Passwords adhere to compliance requirements etc

Be sure to document your test procedure(s) step-by-step as well as the test results. Note if the outcome of the test was expected or not. If there is anything that fails during your testing, you need to fix those issues and retest. Don’t skimp on testing…hackers are not forgiving and just like in dodge ball, there are no “do-overs”.

Maintaining

Once you have tested everything and are assured that your organization is where they need to be, you now need to create and maintain a testing program. Don’t try creating a maintenance program prior to everything being tested, as you will surely be making changes to the maintenance program, making are previous efforts null. Your maintenance program needs to have firm dates / times set for scheduled maintenance. You need to have multiple maintenance programs set up such as:

  • Patch management
  • Password management
  • Network account management
  • System management
  • Applications management
  • Operating system management
  • Security administration etc

By setting up multiple maintenance programs you are able to create “silo’s” for each area and assign personnel who are responsible for each of these areas. This allows for a better view should there be a failure in any of these areas…and makes it easier to see where the failure occurred and to fix the area faster.

Worth Considering

There are a few tricks that you can implement on your network that will make a hacker think twice about trying anything. The more difficult you make it for the hacker to attack, the more likely it is that they will go somewhere else to attack. As someone who has spent the better part of the past quarter of a century protecting companies against attackers, I have listed a few neat tricks you can implement:

Honey Pots

A honey pot is a trap set to detect, deflect, or in some manner counteract attempts at unauthorized use of information systems. Generally it consists of a computer, data, or a network site that appears to be part of a network, but is actually isolated, (un)protected, and monitored, and which seems to contain information or a resource of value to attackers. These honey pots can be used to track and in some cases trap and report a hacker.

Trace Routing

Having the attacker’s IP is all well and good, but what can you do with it? The answer is, a lot more! It’s not enough to have the address, you also need to know where the attacker’s connections are coming from. You may have used automated trace routing tools before, but do you know how they work?

Go back to MSDOS and type tracert *type IP address/hostname here*

Now, what happens is, the Trace route will show you all the computers in between you and the target machine, including blockages, firewalls etc. More often than not, the hostname address listed before the final one will belong to the hacker’s ISP company. It’ll either say who the ISP is somewhere in there, or else you run a second trace on the new IP/hostname address to see who the ISP Company in question is.

Reverse DNS Query

This is probably the most effective way of running a trace on somebody. If ever you’re in a chat room and you see someone saying that they’ve “hacked into a satellite orbiting the Earth, and are taking pictures of your house right now”, ignore them because that’s just bad movie nonsense. THIS method is the way to go, with regard to finding out what country (even maybe what state/city etc.) someone resides, although it’s actually almost impossible to find an EXACT geographical location without actually breaking into your ISP’s head office and running off with the safe.

To run an rDNS query, simply go back to MS-DOS and type netstat and hit return. Any active connections will resolve to hostnames rather than a numerical format.

DNS stands for Domain Name Server. These are machines connected to the Internet whose job it is to keep track of the IP Addresses and Domain Names of other machines. When called upon, they take the ASCII Domain Name and convert it to the relevant numeric IP Address. A DNS search translates a hostname into an IP address….which is why we can enter “www.hotmail.com” and get the website to come up, instead of having to actually remember Hotmail’s IP address and enter that instead.

Well, reverse DNS, of course, translates the IP address into a hostname (i.e., in letters and words instead of numbers, because sometimes the hacker will employ various methods to stop netstat from picking up a correct hostname).

While we’ve given you a very high level look at what needs to be done to better protect yourself from a hack attack, we believe it represents the best place to start in understanding what you need to do.

James Holler is founder of Abidance Consulting.

TwitterFacebookDiggDeliciousTechnorati FavoritesEmailPrintFriendlyShare

Part I: Protect Your Data and Your Company From an Internal or External “Hack-Attack”

James Holler, Founder, Abidance Consulting

Part 1 of a 2-part series

First, let me start with the bad news: There is no absolute way to prevent an internal or external hack-attack. With that said, there are some things that you can do that will make it so difficult to pull off a hack-attack, that the perpetrator will most likely want to go somewhere else and try their attack.

Now, there is an old saying, “cleanliness is next to Godliness.” I am sure you have all heard that line at some time in your life. This saying holds true in the security world. If your network is in total shambles (DAT files not updated, Service Packs are so far behind your need an abacus to determine how many versions behind you are, etc.) and your Intrusion Detection System (IDS) is monitored by humans only during business hours, then you have a “dirty” network that needs to either be cleaned, or as my mom used to tell me…let’s just burn your room and start over, it will be easier that way. If your network/server room looks as if a spaghetti factory has blown up, get it cleaned up by rewiring it using tags on each line so you know where each of the cables is assigned.

The first thing you need to understand in preparing to get your network in top form is to not only determine what is wrong with it, but to also be open to criticism from experts. Put away the ego (one of the top reasons why networks are in shambles to begin with) so that you can listen and learn from your internal experts or external consultants – you hired them, now listen to them.

In Part 1, we’ll look at network discovery issues, vulnerability assessments, and discuss ways to fix some of these challenges.

Network Discovery

Before you can determine what’s wrong with your network, you must first know what your network looks like. You will want to conduct a thorough network discovery since you are going to need to know not only what devices are on your network, but also where they are. Please don’t think that you are going to run a piece of software that will show you everything. If you have a wireless or dial-up modem hanging off of your network and the power button is off, you may never discover it. You may need to do a physical inspection of your entire facility…look up in the ceiling…those pesky tiles can support the weight of a modem and even an old sandwich from 4 years ago. I personally use an iPaq handheld device that is capable of “sniffing” out these modems, even when they are turned off. Now that you have a true and correct picture of your network, you will need to conduct a vulnerability assessment to determine what areas are weak and are in need of attention.

Vulnerability Assessment

To ensure that there are no “cover-ups” by your staff, it is recommended that you have an outside consulting firm come in and conduct the assessment for you. Depending on the size of your organization, the fee’s for this could be $15k to $30k or more. The final report to be delivered should be comprehensive in nature. Be sure to ask for sample reports prior to awarding a contract or project to anyone. There are areas that must be looked at closely. Make sure whoever you assign the project to gives you a list of the services they are going to run. My only word of caution here is that you do not allow a penetration attack be made against your Primary Domain Controller (PDC). Once the assessment is completed, make sure that you not only address the issues, but fix the issues.

Fixing The Issues

When you do get the final report, there are going to be a lot of errors that need to be fixed. Don’t worry; the “bark” of the report is much worse than the “bite”. Depending on how bad your network was when the assessment was conducted, you may have a few pages of issues to as much as a thousand pages of issues – one assessment we did a few years back yielded almost 7,000 pages (a government agency…need I say more). When you are reading your final report, one of the first questions you need to ask yourself is, “Where do I begin”? Not to worry, your security staff/consultants should prioritize what needs to be done and at what point in the project does it need to be done. The point at which a certain task is completed is very important since everything has a logical order of semblance to it…you wouldn’t put the seats in a car before you laid down the carpet. Your staff and/or consultants should know this and be able to build out a project plan with a scope of work, keeping you (the stakeholder) in the loop at all times. Never be afraid to ask questions or challenge something if you feel it isn’t the right thing to do or you don’t understand why something is or isn’t being done.

To save time and money, you have to look at all of the different compliance issues you have to deal with (NERC, EPA, OSHA etc) and cross-walk your efforts to all of these compliance requirements. Doing this will ultimately save yourself time and money by not overlapping efforts.

Next time, we’ll look at testing, maintaining, and some other important issues that merit your attention.

James Holler is founder of Abidance Consulting.

TwitterFacebookDiggDeliciousTechnorati FavoritesEmailPrintFriendlyShare

Hackers Up the Ante in Attack on Electronic Data in Power Plants and Other Facilities

James Holler, Founder, Abidance Consulting

According to the Wall Street Journal (WSJ), computer hackers have designed a virus that targets the industrial control systems, to include power plants, built by German engineering giant Siemens AG. The virus apparently activates a kind of malicious software that analysts say represents a growing corporate-espionage threat. This type of threat has been talked about for years — and it is now a reality.

The virus, Stuxnet, is spread by USB devices plugged into the physically unsecured USB ports on the machine(s) hosting the SCADA systems used by power plants and other types of facilities. The virus is programmed to steal data from computer systems that are used to monitor power plants built for anything from manufacturing to power generation to water treatment.

Researchers analyzing the virus say that they are now seeing several thousand infection attempts daily, though the virus is only activated if it lands on a computer running the Siemens systems software. Analysts warn that the attack on the Siemens’s systems marks an escalation in hackers’ efforts to use viruses for industrial espionage or sabotage purposes. This attack will surely make the NERC CIP regulations become even tighter more quickly than before this story broke.

Smaller, more isolated virus attacks have been attempted before on SCADA systems, but this is the first such infection where a virus is searching specifically for SCADA systems to attack on such a large-scale basis. The worry among security analysts should be that such viruses will, at some point, be used by criminal organizations or even terror groups to sabotage power plants.

The Stuxnet virus specifically exploits an unpatched vulnerability in the Microsoft Windows operating system, allowing it to spread through all USB devices. Once the virus has infected the Siemens system, it uses default passwords that are hard-coded into the Siemens software to upload false control-system data to a remote server. In an advisory that Siemens posted on its website, the company said Microsoft was working on a patch to fix the vulnerability at the USB interface. In its own website advisory, Microsoft has provided a workaround fix to offer some additional protection until a patch, or update, is ready.

Siemens said it expects to approve the updated virus scanners this week and also plans to provide customers with a diagnostic tool to check if their systems have been infected. In the meantime, the company’s website advisory urges customers not to use any USB storage sticks.

Siemens, Microsoft and other security analysts haven’t determined where the virus originated. Many of the infection attempts have originated from India, Indonesia and Iran. The virus likely was created in Asia, given the pattern of attacks and technology used.

James Holler is founder of Abidance Consulting.

TwitterFacebookDiggDeliciousTechnorati FavoritesEmailPrintFriendlyShare

A Weak NERC CIP Program Can Cost You More Than Just A Fine

James Holler, Founder, Abidance Consulting

With more and more emphasis being placed on the CIP requirements, some NERC registered entities may be tempted to “relax” and decide that  they are not deemed classified as a Critical Asset. A word of advice: Take a deep breath and think carefully. If you think that it is just too hard to get compliant and the easiest solution is to just declare that you are not critical, you are very mistaken.

Several attorneys that work in the NERC space have commented that those who deliberately or appear to have otherwise deemed themselves to not be critical simply to avoid having to comply, will most likely face egregious fines from NERC…and we’re talking in the millions of dollars. On the same side of this coin, if your CIP program is so weak that you are judged to not be in compliance, you could suffer much larger financial losses, in addition to any fines, if you are the victim of a cyber attack or an interruption due to your failure to comply.

NERC and FERC have made it very clear from the beginning that they are serious about having registered entities comply with their rules and regulations. Since June 2007, NERC fines total more than $35 million and the FERC fines are almost $120 million.

If your organization doesn’t have the appropriate staff on hand to get the job done, then you have two valid options: 1) hire the appropriate staff, or  2) hire a competent consulting firm. Don’t think that you, as a registered entity, are going to be able to “slip one by” the auditor. The eight RRO’s as well as NERC and FERC have hired on some very skilled cyber security auditors that know what to look for and also where to look. These auditors are very good and the only way to “beat them” is to have a great CIP compliance program in place.

You may ask, “how long does it take to get compliant…and for how much”. This is not an easy question to answer as there are numerous variables that determine the answers to these questions. A few variables are…what is your current state of readiness; have you leveraged from other compliance effort areas such as Sarbanes-Oxley, HIPPA and PCI; have you tested your current state of readiness against a mock audit? There are dozens of factors that must be considered before you can even guess at the timeframe and associated costs. One thing is for sure though…it will cost you a lot less money to get compliant than it will for you to “keep your head in the sand”.

James Holler is founder of Abidance Consulting.

TwitterFacebookDiggDeliciousTechnorati FavoritesEmailPrintFriendlyShare

Telling the Whole Story About US Cyber Attack Program

James Holler, Founder, Abidance Consulting

James Holler, Founder, Abidance Consulting

A recent Wall Street Journal article (July 7, 2010, “U.S. Plans Cyber Shield for Utilities, Companies”) did a good job telling some of the story about this important, and potentially chilling, American initiative. However, I feel the journalist could have, and should have, gone farther with the article. This blog will deconstruct the article and add some important perspective.

First off, I want readers of this blog to understand that I have worked for many government agencies in and around cyber security and was one of the many team members that helped to create the FBI’s cyber snooping system called Carnivore.

Let’s look at some key sections of the article, followed by my thoughts and comments:

  • 2nd paragraph – “The surveillance by the National Security Agency, the government’s chief eavesdropping agency, would rely on a set of sensors deployed in computer networks for critical infrastructure that would be triggered by unusual activity suggesting an impending cyber attack, though it wouldn’t persistently monitor the whole system, these people said.” The vast majority of networks in this country already have monitoring systems implemented that monitor for unusual activities. Compliance requirements such as FERC 706, PCI, CFATS, HIPAA and many others require these monitoring devices. Devices such as Host-Based Intrusion Detection systems (HID’s) and Network Intrusion Detection systems (NID’s) are on most, if not all, networks in this country, so there is no need for the NSA to implement these items.
  • 6th paragraph – “The overall purpose of the [program] is our Government…feel[s] that they need to insure the Public Sector is doing all they can to secure Infrastructure critical to our National Security.” Raytheon secured an initial $100 million contract for this project but never stated that it was a good idea…only that the Government wants to ensure that the critical infrastructure is protected…but I suggest an audit would do this. Maybe that’s why FERC and NERC are requiring audits to make sure registered entities are securing their networks and critical assets.
  • 8th paragraph – “A U.S. military official called the program long overdue and said any intrusion into privacy is no greater than what the public already endures from traffic cameras. It’s a logical extension of the work federal agencies have done in the past to protect physical attacks on critical infrastructure that could sabotage the government or key parts of the country, the official said.” The fact that the military compares snooping on a company’s network which would give them access to payroll, financial and other sensitive information that could be used against them for the benefit of a politically motivated attack to a traffic camera is just plain silly. The military also says it is pertinent so as to prevent the physical attack on the critical infrastructure…I fail to see how snooping on a corporate network has anything to do with protecting from a physical attack.
  • 9th paragraph – “U.S. intelligence officials have grown increasingly alarmed about what they believe to be Chinese and Russian surveillance of computer systems that control the electric grid and other U.S. infrastructure. Officials are unable to describe the full scope of the problem, however, because they have had limited ability to pull together all the private data.” The reason that the U.S. Intelligence Officials can’t describe the problem because they have had limited abilities is exactly right. However, the way this paragraph is worded would make it seem that the limited abilities are because companies are not cooperating. The truth is that there are too many opinions on how this should be done…including from those who have no idea of what they are doing or saying (politicians). The Government needs to hire a group of hackers like the Chaos Computer Club, Brazil Boys or Masters of Deception to come in and solve these problems. What? You have never heard of these guys? There’s a reason for that. The best guys/gals are never caught, therefore, they are not widely known. Companies like McAfee and Symantec keep dozens of hackers on staff to fight against viruses.
  • 13th paragraph – “With the growth in concern about cyber attacks, these relationships began to extend into the electronic arena, and the only U.S. agency equipped to manage electronic assessments of critical-infrastructure vulnerabilities is the NSA, government and industry officials said.” Are you kidding me? The NSA and many other agencies to include the CIA, FBI, NASA and most other government agencies have been successfully hacked so many times that this argument has more holes in it than a slice of Swiss cheese. These people can’t protect against “60 Minutes” reporters from obtaining sensitive information, how in the world can they protect against a cyber terrorist?
  • The article states in the 2nd to last paragraph – “While the government can’t force companies to work with it, it can provide incentives to urge them to cooperate, particularly if the government already buys services from that company, officials said.” Personally, I always get a little nervous when a regulatory body talks about incentives. That is Government speak for “Do what we say or the President will invoke an emergency on your facility under the GRID Act and take your facility from you.

The “Perfect Citizen” project is, in my opinion, just one more way the politicians will attempt to grab control of a private company all in the name of national security. There are only two groups who will benefit from this – Government agencies and consulting firms like Abidance Consulting. The Government needs to take a few steps back and reassess their position. A good recommendation would be to complete audits on all critical infrastructure facilities and determine their state of readiness for a cyber attack based on best practices created by organizations such as NIST 800-53 or ISO-17799. After the audits have been conducted, the Government should issue “warning citations” stating, in detail, what the shortcomings are of that facility and to give them an opportunity to make amends. If they fail to comply, then implement stronger measures against them.

By doing this, the Government will make friends, keep friends and will ensure that companies will do what they need to do for fear that they could lose everything. If you just come right out and force this on a company, there is no incentive on their part to cooperate.

James Holler is founder of Abidance Consulting.

TwitterFacebookDiggDeliciousTechnorati FavoritesEmailPrintFriendlyShare

Savvy Compliance Strategy Part II – Checking Compliance

 

Sal Lucido, VP Enterprise Solutions, AssurX

Sal Lucido, VP Enterprise Solutions, AssurX

In Part I, we took a high-level look at a process for automating regulatory compliance management. The closed-loop process starts with Documenting your processes followed by Monitoring or Checking that your processes are being followed. Next you provide a means of Logging or Tracking any problems that may arise and then take actions to Improve. This improvement should then result in a revision to the Documented process followed by notifying or training those affected by the process improvement.  This closed-loop process, which I call the Circle of Compliance, should be used to automate the process of complying with regulatory standards.

 

The Circle of Compliance

The Circle of Compliance

 

Now lets take a closer look at the Check step. The goal of this step is to eliminate the need to manually audit a process in order to determine its effectiveness. One way to do this is by defining a Key Performance Indicator (KPI). That’s a measure of performance that is used to help an organization monitor progress to goals. For example, a company may decide to improve responsiveness by reducing the number of late tasks. A company might also set a goal for reducing violations or incidents to improve conformance to regulations or standards. You can see an example dashboard showing these two KPI’s in the diagram shown below.

Key Performance Indicators for monitoring late tasks and monthly incidents. Traffic Light indicators provide a method for quickly showing progress to goals

Key Performance Indicators for monitoring late tasks and monthly incidents. Traffic Light indicators provide a method for quickly showing progress to goals

Key Performance Indicators for monitoring late tasks and monthly incidents. Traffic Light indicators provide a method for quickly showing progress to goals.

Let’s take a closer look at this KPI dashboard. Both measurements are listed: Late Projects and Monthly Incidents. Notice that the date the measurement was made along with the actual performance data are displayed. We can see that for the month of May there were two late projects and five incidents. Then on the right we see a trend arrow (more on this below) and a traffic light, which give us a quick indication of performance to goal. Green is good and red is bad. Of course in order to set the traffic light to the correct state (green, yellow or red) we need some goals.

For example if there are less than two late projects each month the light will be green. If there are between two and four late projects we would consider that a yellow light (or caution). And if there were more than four late projects in a given month we would set the light to red.

When implemented properly, KPI’s monitor performance over a given time period (day, week, month, etc.) and provide a visual indication (traffic light, flag, etc.) of performance to goal. So let’s dig a bit deeper to better understand how to do it right.

Since a KPI measures performance over a given time period there must be historical data, trends and state changes. Let’s start with historical data. By clicking on the KPI dashboard we can see past measurements (shown below).

A report of historical KPI data shows an improving trend. An email is automatically sent in May when the light changes state.

A report of historical KPI data shows an improving trend. An email is automatically sent in May when the light changes state.

A report of historical KPI data shows an improving trend. An email is automatically sent in May when the light changes state.

We can see from the historical data that the trend is moving from bad to good and that in May there was a state change to red and yellow respectively. This system is set up to automatically send an email to the KPI Owner whenever there is a state change.

Emails are automatically sent when the light changes state. This shows a notification indicated that a things are getting worse given the light changed from green to yellow.

Emails are automatically sent when the light changes state. This shows a notification indicated that a things are getting worse given the light changed from green to yellow.

Emails are automatically sent when the light changes state. This shows a notification indicated that things are getting worse given the light changed from green to yellow.

Also if you look back at the KPI Dashboard you see the Trend arrow is green and down. Down indicates that we have fewer late projects than in the previous reporting period. The arrow is green, which indicates that this is a ‘good’ or desirable trend.

In summary, setting up Key Performance Indicators that monitor your performance to goals is a good way to ‘Check’ that your processes are working properly. It also eliminates the need to perform manual audits of
a given operation reducing labor costs. The next step in this closed-loop process is ‘Tracking Problems’.

Next time: We’ll take an in depth look at the ‘Tracking Problems’ step.

Read Read Part IIII and IV.

Sal Lucido is Vice President, Enterprise Solutions at AssurX, Inc. You can follow him at http://twitter.com/ComplianceTips

TwitterFacebookDiggDeliciousTechnorati FavoritesEmailPrintFriendlyShare

Savvy Compliance Strategy Can Ease GMP, Electric Reliability Regulatory Challenges

Sal Lucido, VP Enterprise Solutions, AssurX

Sal Lucido, VP Enterprise Solutions, AssurX

The primary function of the compliance department is to ensure that the company complies with all of the applicable regulations, rules, and laws. Regardless of industry (life science, energy and utilities, financial services, etc.) this is a universal charter.

As someone who serves customers across many heavily regulated industries, I think I’ve got a unique perspective — and I’d like to share some of what I’ve learned along the way in the hopes that it helps you in some small way .

One particularly useful tool I see used across all industries is what I call the ‘Circle of Compliance’. Before I explain this concept, let’s take a deeper look at the job of the compliance department.

As I’ve already mentioned, the compliance department is put in charge of ensuring that all applicable compliance requirements are met. For example U.S. medical device companies must comply with the FDA’s Good Manufacturing Practices (GMP). Regulation 21 CFR Part 820.90 states that each manufacturer shall establish and maintain procedures to control product that does not conform to specified requirements. So the compliance department must determine if their company follows this process.

This is not so different from a U.S. power company that owns transmission lines. They must comply with Reliability Standard FAC-003 that mandates a clearance be maintained between transmission lines and vegetation. It also requires the company to report any vegetation related outages. These are different industries and different regulators (FDA, NERC), but each has the same fundamental task.

So how does the compliance department go about ensuring these regulations are met? Typically they audit the company for compliance. If there is a gap between the requirement and current practice, they work with the appropriate departments to close the gap. Take a look at this illustration for a visual representation of this ‘push’ exercise.

Relying on the Compliance Department to close compliance gaps is a time consuming, never-ending job…

Relying on the Compliance Department to close compliance gaps is a time consuming, never-ending job…

You can see from the illustration that this is a manual task. The problem is that it is a time consuming, never-ending job. As soon as the compliance department shifts their attention to another area of the company, compliance gaps can (and usually do) reappear. This is then addressed with ‘periodic’ audits. What we end up with is an endless and expensive merry-go-round of audits and fixes.

The solution? Set up a process that continuously ‘pulls’ the operations towards the regulations. I’ve illustrated this type of system below.

…it is better to implement processes that automatically and continuously close compliance gaps.

…it is better to implement processes that automatically and continuously close compliance gaps.

You can see the advantage of this system from the illustration. It does not require the constant and repeated attention of the compliance department.

So what is this process? I call it the ‘Circle of Compliance’ as illustrated below.

The Circle of Compliance

The Circle of Compliance

In a nutshell, this is a closed-loop corrective/preventive action process. While you might recognize the process as it relates to quality systems, you may not have considered its application to the job of regulatory compliance.

This is how the process works: Let’s look at the U.S. power company that must ensure that trees are kept away from transmission lines. Of course the compliance group would first check to make sure the vegetation inspection and removal procedure is ‘Documented’ adequately.

Next the compliance group would see if there is a ‘system’ in place for monitoring that the process remains effective. This is the ‘Check’ part of the process. Also they would ensure that there is a process for documenting problems such as vegetation related outages. Most compliance departments do a good job of auditing these two steps, but it is crucial that the next two steps are completed.

Any and all problems with the vegetation monitoring system must be ‘Tracked’. This means they must be documented in a system that links directly to the next step: Improve. All problems must be looked at to determine how the problem occurred and how the system can be ‘Improved’ to prevent reoccurrence. This improvement must then result in a change to the ‘Documented’ process followed by retraining of the workforce to the new process.

If implemented properly this closed-loop ‘Circle of Compliance’ will save the company time and money while improving its ability to comply with industry regulations.

Next time: I’ll explore each of these steps (Document, Check, Track and Improve) in more detail.

Read Part II, III and IV.

Sal Lucido is Vice President, Enterprise Solutions at AssurX, Inc. You can follow him at http://twitter.com/ComplianceTips

 

 

 

 

TwitterFacebookDiggDeliciousTechnorati FavoritesEmailPrintFriendlyShare

The Top 10 FERC Enforceable Standards in 2009

We knew it was coming. FERC enforcement actions picked up significantly in 2009. But what surprised us most was the eye-popping 214 violations for PRC-005-1.

Top 10 FERC Enforceable Standards of 2009

So, what exactly is PRC-005-1?

Simply put, the NERC PRC-005-1 standard says (but is not limited to the following – see below for link/download of standard):

  • Owners of generation Protection Systems shall have a maintenance and testing program for all protection devices that affect the reliability of the Bulk Electric System (BES).
  • The program must include a maintenance and testing interval for each type of device and the basis used for determining that interval.
  • The owner must provide documentation of its system maintenance and testing program and the implementation of that program to its Regional Reliability Organization on request within 30 calendar days.
  • The program must provide the evidence that each Protection System device was maintained and tested within the defined intervals including the last tested/maintained date.

You can download the NERC PRC-005-1 Standard here.

Why has complying with PRC-005-1 been such a challenge to utilities?

The legacy of the unregulated past of both small and large utilities has left them with islands of disconnected data related to physical devices and assets that have an affect on the BES. This data consists primarily of work orders, maintenance activities, test results and supporting evidentiary documentation. Unfortunately, all of this data tends to reside in disparate systems such as ERP, spreadsheets, hard copies, custom databases, test systems, etc., and are typically spread across various departments and facilities in the enterprise. If a utility is found in non-compliance with the NERC Standard during an audit, it can result in fines from the tens to hundreds of thousands of dollars.

Achieving compliance with PRC-005-1

As daunting a challenge this might sound, there is a cost effective and simple solution that provides a central hub for real-time visual compliance to NERC PRC Standards. CATSWeb ER seamlessly integrates data from dozens of differing sources into a single repository and creates logical relationships amongst this data to perform measurements and analytics in real-time dashboards. This provides a visual status of PRC compliance of all devices and assets across the enterprise. Users also get real-time alerts and notifications as trends towards non-compliance occur, well before a state of non-compliance happens. The system also maintains a complete auditable history that’s ready for an audit of all devices, issues and actions in maintaining control and compliance.

Visit the NERC website to view enforcement actions, settlements and penalties.

TwitterFacebookDiggDeliciousTechnorati FavoritesEmailPrintFriendlyShare

NERC Policy Statement Powered with Common Sense, Clarity

electricitylightbulbWe’ve got some slightly bad news and some really good news.

The bad news is you only have until April 19 to file your comments on NERC’s just-issued Bulk Power System Critical Infrastructure Policy Statement.

The good news:  It is so clear and grounded in common sense that your only comment may be “nice job.”

Reminds me of what baseball players and managers say about umpiring: We can live with almost any reasonable interpretation of the strike zone, we just want clarity and consistency in your calls.

Well, NERC seems to have accomplished that – at least in the big picture sense.

“Yes, it is high level,” says AssurX NERC expert Paul Fricke, but it lays “good ground work.” NERC’s approach here “makes me feel more secure,” Paul adds. “The policy statement is absolutely grounded in common sense.”

Specifically, it mandates that NERC and its members address cyber security, physical security, and other high impact threats.

The statement calls for a multi-element strategy that addresses asset prioritization, risk information management, standards, prevention and detection, resilience, readiness, response, restoration, roles and authorities, communications, evaluation and testing, technical studies, interdependencies and funding.

NERC’s new umbrella statement may be in response to the heat the Department of Homeland Security and other government officials have publicly put on NERC, FERC and other keepers of the infrastcture. Their concern: The nation’s vital systems are still vulnerable to threats, both internal (natural disaster, for example) or external (think terrorism). Or in NERC-speak, “A significant concern is the potential for disruptions impacting large portions of the bulk power system, whether by intentional attack or natural event, from which restoration and recovery may be challenging and prolonged.”

Fricke believes the relatively short comment period is also a result of that pressure. “DHS is very mindful of the threats to the power grid and they want companies to show they have controls and checks in place to protect high-priority assets in particular.”

We don’t expect there to be any earth-shattering comments from industry, but we’ll keep you posted.

TwitterFacebookDiggDeliciousTechnorati FavoritesEmailPrintFriendlyShare

FDA Lifts Curtain on Inspection Process, Rationale

Michael Causey, Editor & Publisher, eDataIntegrityReport.com

Michael Causey, Editor & Publisher, eDataIntegrityReport.com

Gotta give the FDA some credit here. In addition to its transparency initiative we’ve talk about before, the agency is also trying to remove some of the mystery about how it handles inspections and other inner workings at the FDA. From where I sit, it appears to be a sincere effort and I believe it is helping outsiders better understand what the FDA is trying to do – and how it is trying to do it.

For example, at the second in a new series of monthly online webinars, FDA’s Michael C. Rogers, deputy director, Office of Regional Operations, said today (March 25, 2010)  tried to outline how an FDA inspection tends to work, and what drives inspectors before, during and after an inspection.

As an aside, Rogers also said that the agency currently has about 1,800 total inspectors across its full portfolio, though food gets the bulk of the bodies. He also said there will be more foreign inspections this year, and that the number should continue to grow.

Inspections are based on risk, Rogers said. In other words, the riskier the potential drug, device or food item, the more likely they will be inspected.

Most inspections are unannounced, Rogers said. Before they go on-site, the inspector on inspection team will look at previous inspection reports and identify what corrective actions were promised during prior inspections. They also prepare inspection tool kits with sampling equipment, info to drive inspection based on guidance documents and the Investigation Operations Manual. They also carry a camera to document evidence.

They also conduct “for cause” inspections driven by consumer complaints or other outside activity.

Typically, the inspection begins with a discussion with management to explain the purpose of the inspection, and they try to learn about the corporate structure and any changes made since last inspection. They also ask about complaints, positive tests or returns. Answers to those questions help FDA inspectors focus their on-site efforts.

Next, they go to the physical manufacturing area. They try to observe and understand the on-site process. They ask about acceptance criteria and want specifics on failures, especially the reasons.

Inspectors also draw a diagram of the facility showing the manufacturing process from start to finish. They’re looking for problems in the system and looking to identify critical control points in the manufacturing process.

FDA inspectors then identify procedures in place and assess if company is actually following them. They also look for controls in place to mitigate any contaminated products.

They also look at training and cleaning programs. They also watch employees while they are actually making the product.

If they find evidence of an adulterated product, they collect evidence based on inspector observations and collect samples to prepare their case for possible legal action in court.

At conclusion of inspection, the FDA team meets again with management. They then inform the top company official what is in the official Form 483. That form documents observations during the inspection but does not include final recommendations. They also ask for the firms corrective actions planned or in place to get into compliance.

These corrective actions are taken into account as agency formulates official recommendations.

After the inspection at the firm, the inspector develops a report back at the home office. It includes evidence collected and what the firm has already agreed to do about any shortcomings.

In some cases firms can offer voluntary corrections. But sometimes the agency decides it needs enforcement action such as a warning letter, and can also impose civil and/or monetary penalties.

The webinar was extremely popular. In fact, it “sold out” so many who tried to join it could not get in to the live event. There will be a recording available on Monday March 29.

UPDATE: Slides are now available from this event here in PDF format.

TwitterFacebookDiggDeliciousTechnorati FavoritesEmailPrintFriendlyShare