I woke up yesterday feeling a bit ragged around the edges – sneezing / sniffing, coughing, eyes watering. No it’s not flu (not even Man flu) just a bit of a later summer cold. No aches, pains, shivering etc.
However, it did make me think again about our Business Continuity planning. I did some work on this a couple of years ago and we actually have a basic outline plan of what to do. I went through it earlier in the year with the HR manager as they were concerned about the possibilty of the swine flu epidemic causing us some issues.
Although nothing much happened at the time, going on past experience of influenza pandemics, there will be another bout of it over the winter months, and possibly a third wave later next year. Although it seems that swine flu is not as virulent as other strains, it could still cause some staff to take time off work.
As we are a manufacturer, production is likely to be affected by a reduction in staff available. They do have the capacity to run some light shifts, but it may be necessary to move production between the different sites. Fortunately possible as we have standardised methods – we don’t yet have a single system to control this, but that is a longer term project that is underway.
For office staff, we currently have a number of key people that work with laptops; these people have a remote access facilty that allows them to connect to internal systems and continue working. We are currently looking to replace a number of those laptops and the older ones will be kept back, cleaned down and issued for use in an emergency situation.
We also installed a new telephone systems a while back – we still have some more modules that could be added to this, but we hope the new system will aslo improve the way that staff can communicate with customers and suppliers in the event of some issues.
But one item that I did discuss with management hasn’t been fully implemented yet – a scheme to encourage better hygienic practices. Anyone that has worked in food production or in the health service will know that considerable resources go to training people to take more care over preventing passing on germs. I worked with some people in a lab for a while and the level of bacterial infection found in some of the swab tests could really put you off your lunch! And I’m sure everyone has stories of what they have found under the keys of a keyboard!
It’s strange how people that are so careful to wash their hands after relieving them selves, don’t bother to give their work surface even the most cursory of cleaning. And how many telephone handsets get a wipe with a cloth that looks dirtier than the surface it’s being used to clean? Perhaps we need to have a new certification – the correct sanitisation and cleaning of Personal Computers!
Showing posts with label BCP. Show all posts
Showing posts with label BCP. Show all posts
Monday, 21 September 2009
Tuesday, 11 August 2009
The battle of the Vs – VMWare vs Hyper V
Last week, I and one of the guys in the department had the chance to attend an event that demoed VMWare 4 and Windows Server 2008 R2 with Hyper-V. Many thanks to the people at Nexus in Exeter (http://www.nexusopensystems.co.uk/) that hosted the event, especially Gary that did the demos.
The presentations were quite straight forward. The first session was VMWare and we were shown the installation process starting right from the bare metal server. The actual installation process was Linux based and took about 20 minutes on releatively low spec machines. Once it was all up and running, we had the chance to see some virtual servers being created – literally just a matter of a few minutes work. We also discussed the switching process and the various options, and briefly saw how to create virtual switches.
There was a bit of a discussion about the merits of the VMWare product – how it allows you to “overload” by selecting options for the virtual servers such as levels of RAM that total up to more than the physical amount actually available. I would want to check this out for myself, but it certainly seemed to run OK.
We then discussed clustering and resiliance and the demo that followed showed a high definition media file being moved from one virtual server to another – the file ran constantly during the move and there was not even a slight pause during the process. Really impressive! Certainly, this would be of significant value in a situation where you are having to move production data when people are still working on it.
The demo actually ran over a bit as we were really interested in the product and had several questions to ask about various aspects – and Gary was only too pleased to show us the various bits in response. There is no question that it is an awesome product.
We then had the chance to see Hyper-V in action and for me it was the first chance I’ve had to look at this. We have Windows Server 2008, but not the R2 version which contains the hypervisor. The main difference between the two is that the VMWare hypervisor sits above the hardware and handles all of the driver requirements. Hyper V sits at the same level as the OS, just above the hardware, but each virtual server will handle it’s own drivers seperately. It also doesn’t allow overloading of resources – once you hit the limit, that’s it.
From what we saw, the Hyper V runs well – certainly it provided a smooth experience whilst we were watching it and the test moving the media file ran pretty much the same. There were a few diffences in the way that the virtual networking operates, but certainly it seems to run as we had expected. It definitely doesn’t have all the functionality of VMWare, but then there is a price difference – it’s a lot cheaper.
I’ve been looking at this now for a few months (in between other jobs) and I’m convinced that virtualisation is the way to go. It will certainly cut costs in terms of the electric bill, and it will also fit very nicely into our backup process / business continuity / disaster recovery planning. About half of our servers will reach 5 years old next year, so it seems a good time to start planning a move over to a virtualised system.
We have had a couple of visits to different vendor demos and they have been really useful. Although nothing has been decided, we are leaning towards the Dell Equalogic equipment – it seems to be everything that we could want and a bit more. The big issue of course is what software to run on the servers which is why we wanted to get to the event in Exeter. Howevever, I still not sure which one I think is the best option for us.
I’ve therefore planned that in the new year, say Jan / Feb 2010, we will get ourselves a spare server – there are plenty of cheap machines around at the moment. There is a trial version of VMWare available and of course, the Technet subscription allows us to install an evaluation copy of Server 2008 R2. Hopefully, this will gives us the chance to actually work with both products so that we can get a really good idea of which one we prefer – all we then have to do then is sell it to the powers that be!
The presentations were quite straight forward. The first session was VMWare and we were shown the installation process starting right from the bare metal server. The actual installation process was Linux based and took about 20 minutes on releatively low spec machines. Once it was all up and running, we had the chance to see some virtual servers being created – literally just a matter of a few minutes work. We also discussed the switching process and the various options, and briefly saw how to create virtual switches.
There was a bit of a discussion about the merits of the VMWare product – how it allows you to “overload” by selecting options for the virtual servers such as levels of RAM that total up to more than the physical amount actually available. I would want to check this out for myself, but it certainly seemed to run OK.
We then discussed clustering and resiliance and the demo that followed showed a high definition media file being moved from one virtual server to another – the file ran constantly during the move and there was not even a slight pause during the process. Really impressive! Certainly, this would be of significant value in a situation where you are having to move production data when people are still working on it.
The demo actually ran over a bit as we were really interested in the product and had several questions to ask about various aspects – and Gary was only too pleased to show us the various bits in response. There is no question that it is an awesome product.
We then had the chance to see Hyper-V in action and for me it was the first chance I’ve had to look at this. We have Windows Server 2008, but not the R2 version which contains the hypervisor. The main difference between the two is that the VMWare hypervisor sits above the hardware and handles all of the driver requirements. Hyper V sits at the same level as the OS, just above the hardware, but each virtual server will handle it’s own drivers seperately. It also doesn’t allow overloading of resources – once you hit the limit, that’s it.
From what we saw, the Hyper V runs well – certainly it provided a smooth experience whilst we were watching it and the test moving the media file ran pretty much the same. There were a few diffences in the way that the virtual networking operates, but certainly it seems to run as we had expected. It definitely doesn’t have all the functionality of VMWare, but then there is a price difference – it’s a lot cheaper.
I’ve been looking at this now for a few months (in between other jobs) and I’m convinced that virtualisation is the way to go. It will certainly cut costs in terms of the electric bill, and it will also fit very nicely into our backup process / business continuity / disaster recovery planning. About half of our servers will reach 5 years old next year, so it seems a good time to start planning a move over to a virtualised system.
We have had a couple of visits to different vendor demos and they have been really useful. Although nothing has been decided, we are leaning towards the Dell Equalogic equipment – it seems to be everything that we could want and a bit more. The big issue of course is what software to run on the servers which is why we wanted to get to the event in Exeter. Howevever, I still not sure which one I think is the best option for us.
I’ve therefore planned that in the new year, say Jan / Feb 2010, we will get ourselves a spare server – there are plenty of cheap machines around at the moment. There is a trial version of VMWare available and of course, the Technet subscription allows us to install an evaluation copy of Server 2008 R2. Hopefully, this will gives us the chance to actually work with both products so that we can get a really good idea of which one we prefer – all we then have to do then is sell it to the powers that be!
Thursday, 14 May 2009
Spam, spam, spam, spam, spam .................
We had a slight problem yesterday – access to the Internet became a bit flaky. People kept calling to say that access was denied or sites were taking a long time to open. At first, we thought that this was just people being impatient, but quickly we realised that there was a problem.
The firewall device seemed to be struggling a bit – the connection log showed a very high level of packet transmission. That wasn’t too unusual as our current connection gets maxed out on a regular basis and we have seen it much worse. A few tweaks and it all seemed OK – so we thought no more of it.
Later in the afternoon, we had the same problem; tweak again, all OK – but then it happened again very quickly afterwards. A brief discussion and it was decided that we should re-boot the device to clear anything that was cached that might cause a problem. One quick reboot and everything was hunky dory.
When I got into the office this morning I was a bit surprised to find about 20 or so emails that were exact duplicates of ones that I had received yesterday. I asked around and a number of other people had the same problem. I did a few checks, but couldn’t see any problems. There seemed to be the usual level of network activity – nothing that would indicate any issues so I put it down to the previous day’s problems.
Over the next couple of hours, I worked on various items including a few support issues. During that time, I received several more emails, some internal, some external. Around mid morning, I thought about it and realised that I had actually received no new external mail, they had all been duplicates. I did a quick check using an external mail service, and realised that there were no incoming or outgoing mails at all.
The guys and I did some tests and quickly realised that something was seriously wrong with the firewall – it was running like a 3 legged dog and several pages of the control menu just would not open at all. We called the support team at the mail service and they checked but confirmed mail was coming in – so we called the vendor of the firewall. They checked but also found that it was running slow so they escalated the problem to the manufacturer.
About an hour later, we got a call from the vendor – the support guys from the manufacturer had found that there were a lot of emails in the cache of the device – about a 1000 or so. They said that they would run a script to clear the cache and expected that this would fix the problem. About 20 minutes later, they phoned back again – it wasn’t a thousand, but one hundred thousand! - and more coming in by the second.
Eventually, they cleared the cache and the email started to move, and our spam mailbox suddenly started to groan under the weight of the mail. It was all from one IP address in Japan, to one mailbox, with one subject line. A quick calculation showed over 10,000 incoming mail every hour. To deal with it, we set-up a PC logged on with the user account for the spam mailbox, and then we set a rule within Outlook to delete the incoming mail from the specific sender. Once this was set running, we could see the incoming mail, but also see it being deleted – it was really cool to watch.
All in all, we feel pretty good about it; once the problem was identified, we had a solution really quickly. Yes we did have a period of a couple of hours with no email, but no-one actually realised this. One of the directors did have an issue with trying to send an important mail to a potential client; but I was able to do that for him using a specific backup external mail facility set-up for that purpose.
After identifying the problem we had outgoing mail within about 15 minutes – incoming mail took slightly longer because of the backlog of garbage, but still less than 30 minutes. The staff were all kept informed – but later it seemed, most of them hadn’t even realised that there was a problem until they got the email from me to tell them about it.
I’m going to sit down with the guys in the next few days – we will draw up a brief outline of what happened and will use that to see if there was anything else we could have done to (1) prevent it, (2) detect it, (3) prepare for it happening again. This will be added to our Business Continuity / Disaster Recovery Plans
The firewall device seemed to be struggling a bit – the connection log showed a very high level of packet transmission. That wasn’t too unusual as our current connection gets maxed out on a regular basis and we have seen it much worse. A few tweaks and it all seemed OK – so we thought no more of it.
Later in the afternoon, we had the same problem; tweak again, all OK – but then it happened again very quickly afterwards. A brief discussion and it was decided that we should re-boot the device to clear anything that was cached that might cause a problem. One quick reboot and everything was hunky dory.
When I got into the office this morning I was a bit surprised to find about 20 or so emails that were exact duplicates of ones that I had received yesterday. I asked around and a number of other people had the same problem. I did a few checks, but couldn’t see any problems. There seemed to be the usual level of network activity – nothing that would indicate any issues so I put it down to the previous day’s problems.
Over the next couple of hours, I worked on various items including a few support issues. During that time, I received several more emails, some internal, some external. Around mid morning, I thought about it and realised that I had actually received no new external mail, they had all been duplicates. I did a quick check using an external mail service, and realised that there were no incoming or outgoing mails at all.
The guys and I did some tests and quickly realised that something was seriously wrong with the firewall – it was running like a 3 legged dog and several pages of the control menu just would not open at all. We called the support team at the mail service and they checked but confirmed mail was coming in – so we called the vendor of the firewall. They checked but also found that it was running slow so they escalated the problem to the manufacturer.
About an hour later, we got a call from the vendor – the support guys from the manufacturer had found that there were a lot of emails in the cache of the device – about a 1000 or so. They said that they would run a script to clear the cache and expected that this would fix the problem. About 20 minutes later, they phoned back again – it wasn’t a thousand, but one hundred thousand! - and more coming in by the second.
Eventually, they cleared the cache and the email started to move, and our spam mailbox suddenly started to groan under the weight of the mail. It was all from one IP address in Japan, to one mailbox, with one subject line. A quick calculation showed over 10,000 incoming mail every hour. To deal with it, we set-up a PC logged on with the user account for the spam mailbox, and then we set a rule within Outlook to delete the incoming mail from the specific sender. Once this was set running, we could see the incoming mail, but also see it being deleted – it was really cool to watch.
All in all, we feel pretty good about it; once the problem was identified, we had a solution really quickly. Yes we did have a period of a couple of hours with no email, but no-one actually realised this. One of the directors did have an issue with trying to send an important mail to a potential client; but I was able to do that for him using a specific backup external mail facility set-up for that purpose.
After identifying the problem we had outgoing mail within about 15 minutes – incoming mail took slightly longer because of the backlog of garbage, but still less than 30 minutes. The staff were all kept informed – but later it seemed, most of them hadn’t even realised that there was a problem until they got the email from me to tell them about it.
I’m going to sit down with the guys in the next few days – we will draw up a brief outline of what happened and will use that to see if there was anything else we could have done to (1) prevent it, (2) detect it, (3) prepare for it happening again. This will be added to our Business Continuity / Disaster Recovery Plans
Subscribe to:
Posts (Atom)
