Category: Planned Maintenance

Jun 07 2024

Network maintenance and change freeze in July

RM have purchased some new hardware and it will be used to replace the existing central firewall clusters. The existing hardware is approaching full capacity so this work obviously needs to go ahead. The new hardware will be the latest model with a lot more capacity and it will be a direct swap out.

The work will involve copying all the firewall config data and then over three nights (2nd, 3rd and 4th July) the platform will be updated. I am assured that it is “low risk” particularly as the same software version will be used on the new hardware. However, if there are problems, it will be straight forward enough to quickly roll back the changes.

Both RM and HFL are keen that this work goes ahead now rather than waiting for the summer holidays. This is because if in the unlikely event that there are issues, they may not present themselves until schools return in September, when there is high traffic.

This event will incur a change freeze meaning that new firewall rules cannot be created from Monday 1st July 9AM through to Friday 5th July 10AM. This will allow RM to copy over the up-to-date firewall config data. With this in mind, over this period we will not be able to create new firewall rules. This includes RM making firewall changes as well as HFL/School IT Support updating access within SafetyNet.

If you have any queries or concerns, please get in touch with HFL Broadband Support and/or Kevin Crawley.

Nov 01 2022

Firewall reboot

Good afternoon,

RM are planning to perform a reboot of the firewall cluster tonight at 10PM. The firewall will be unavailable for a short period and there will be an outage for the services through the firewall.  RM are performing this change to do some diagnostics as suggested by Fortinet support.

Any queries, please get in touch.

Thanks,

Kevin Crawley and the HfL Broadband Team

Apr 21 2022

Firewall Upgrade

Following on from last night’s firewall upgrade, there are unfortunately some issues. There are some schools who have reported issues with their VOIP/SIP deployments and some Google services are also not working as expected. RM senior engineers are investigating.

Thanks,
Kevin Crawley

Apr 07 2022

Planned upgrade of firewall

In the early hours of Thursday 21st April, RM will be carrying out some work to upgrade the firewalls. If this work doesn’t go ahead, the current version of the software on the firewalls will reach end of engineering support with the vendor. This means they will only provide fixes for industry wide critical issues and vulnerabilities. With this in mind, RM need to upgrade the software version to maintain full levels of vendor support.

RM and HfL have discussed this in detail and RM are confident that normal service will continue without any service disruption. RM and HfL will be fully staffed with the relevant resource should anything unforeseen occur. If you have any concerns or queries in relation to this work – either before, or after the upgrade has been carried out, please get in touch with the HfL Broadband Team.

Thanks,

Kevin Crawley
HfL Broadband Lead

Nov 12 2021

Proxy issues?

In the early hours of this morning, RM engineers added some new SafetyNet Load Balancers into service. This was planned maintenance work to ensure the service remains stable. We have received some reports this morning advising us that there are problems browsing the internet today. This is back with RM who are investigating…

I’ll add another update shortly.

Thanks,

Kevin Crawley

Nov 03 2021

Speed issues

We believe this has now been resolved. It looks like there were two simultaneous issues;

  • RM’s transit to Google was being maxed out. RM will be implementing new routers with 100G links to deal with the increased traffic volumes. RM is hoping for this will be completed by the end of the month.
  • There were also six minor DDoS attacks to the proxy farm IPs at the same time of high utilisation of Google traffic. RM is working with a third party company to mitigate these attacks. As they were minor attacks, it didn’t trigger the automatic mitigation system. RM will be working closely with their third party to investigate what went wrong and to make sure this doesn’t happen again.

If there are any further queries, please contact the HfL Broadband Team.