Kevin Crawley

Author's posts

Issues resolved

This was resolved shortly after 8am this morning.

The problem was only the WF1 proxy. RM identified an issue with accepting the WF1 proxy connections on the new Load Balancer but this has now that has been fixed.

As ever, we will ask RM to provide a report advising what improvements can made to avoid future disruption

Thanks,

Kevin Crawley

Proxy issues?

In the early hours of this morning, RM engineers added some new SafetyNet Load Balancers into service. This was planned maintenance work to ensure the service remains stable. We have received some reports this morning advising us that there are problems browsing the internet today. This is back with RM who are investigating…

I’ll add another update shortly.

Thanks,

Kevin Crawley

Speed issues

We believe this has now been resolved. It looks like there were two simultaneous issues;

  • RM’s transit to Google was being maxed out. RM will be implementing new routers with 100G links to deal with the increased traffic volumes. RM is hoping for this will be completed by the end of the month.
  • There were also six minor DDoS attacks to the proxy farm IPs at the same time of high utilisation of Google traffic. RM is working with a third party company to mitigate these attacks. As they were minor attacks, it didn’t trigger the automatic mitigation system. RM will be working closely with their third party to investigate what went wrong and to make sure this doesn’t happen again.

If there are any further queries, please contact the HfL Broadband Team.

Slow internet

Good morning.

We have received multiple reports of slow internet this morning. RM are investigating. Apologies for the disruption.

Thanks,
Kevin Crawley

Possible Google update and reports of slow internet

Yesterday afternoon (18th October) we received some reports of slow internet.

RM suspect that there was a Google update released which resulted in higher than usual volumes of traffic – and this in turn would cause a slow internet experience. Typically, the RM transit (for Google) utilises at a maximum of 8Gbps, but yesterday it went over 10Gbps.

In fact there were three instances yesterday where the transit bandwidth utilisation exceeded the capacity, timescales for which are updated below.

8.48-9.08
11.25-11.40
13.55-14.03

RM is closely monitoring the situation.

Websites that use websockets – Chromebooks

When using Chromebooks, you may experience issues accessing websites that use websockets. In these instances, the browser on the Chromebook attempts to go out to the internet using the SOCKS protocol entry from the proxy settings.

Normally sessions go out to the internet using the top 3 protocols (HTTP and HTTPS especially). Other devices (Windows etc) have the SOCKS protocol blank so it will send the session to the internet via HTTP/HTTPS instead. On Chromebooks if you’ve set a proxy (wf1.thegrid.org.uk etc) via G-Suite it ticks it for all protocols (including SOCKS). However, RM do not support the SOCKS protocol and this makes the outbound connection fail.

There are some high profile websites that use websockets such as SCOMIS, Spotify, and LiveStorm. It may also cause issues when using the remote desktop web client into a LARA server.

There are a couple of fixes!

Option 1) remove the entry from the SOCKS protocol, and leave it blank.
Option 2) schools connect the Chromebooks to a transparent proxy network (typically a 10.* range), instead of the proxied network (172.* range)

If you have any queries, please get in touch with our Service Desk. Thanks