Category: Planned Maintenance

Firewall reboot

Good afternoon,

RM are planning to perform a reboot of the firewall cluster tonight at 10PM. The firewall will be unavailable for a short period and there will be an outage for the services through the firewall.  RM are performing this change to do some diagnostics as suggested by Fortinet support.

Any queries, please get in touch.

Thanks,

Kevin Crawley and the HfL Broadband Team

Firewall Upgrade

Following on from last night’s firewall upgrade, there are unfortunately some issues. There are some schools who have reported issues with their VOIP/SIP deployments and some Google services are also not working as expected. RM senior engineers are investigating.

Thanks,
Kevin Crawley

Planned upgrade of firewall

In the early hours of Thursday 21st April, RM will be carrying out some work to upgrade the firewalls. If this work doesn’t go ahead, the current version of the software on the firewalls will reach end of engineering support with the vendor. This means they will only provide fixes for industry wide critical issues and vulnerabilities. With this in mind, RM need to upgrade the software version to maintain full levels of vendor support.

RM and HfL have discussed this in detail and RM are confident that normal service will continue without any service disruption. RM and HfL will be fully staffed with the relevant resource should anything unforeseen occur. If you have any concerns or queries in relation to this work – either before, or after the upgrade has been carried out, please get in touch with the HfL Broadband Team.

Thanks,

Kevin Crawley
HfL Broadband Lead

Proxy issues?

In the early hours of this morning, RM engineers added some new SafetyNet Load Balancers into service. This was planned maintenance work to ensure the service remains stable. We have received some reports this morning advising us that there are problems browsing the internet today. This is back with RM who are investigating…

I’ll add another update shortly.

Thanks,

Kevin Crawley

Speed issues

We believe this has now been resolved. It looks like there were two simultaneous issues;

  • RM’s transit to Google was being maxed out. RM will be implementing new routers with 100G links to deal with the increased traffic volumes. RM is hoping for this will be completed by the end of the month.
  • There were also six minor DDoS attacks to the proxy farm IPs at the same time of high utilisation of Google traffic. RM is working with a third party company to mitigate these attacks. As they were minor attacks, it didn’t trigger the automatic mitigation system. RM will be working closely with their third party to investigate what went wrong and to make sure this doesn’t happen again.

If there are any further queries, please contact the HfL Broadband Team.

HfL Firewall Maintenance

RM are planning to perform important maintenance to the HfL firewall on 30th August 2019 between 4.00 AM and 7.00 AM UK time. We are not anticipating any service interruptions during this change but should you experience any issues, please contact the Service Desk.

Jenny Collins – Service Desk Manager