ChannelLife New Zealand logo
Industry insider news for New Zealand's technology resellers
Story image

Cloudflare outage in 19 data centers worldwide due to own error

By Jessie Chiang
Wed 22 Jun 2022

Content delivery network Cloudflare says its outage for 19 of its data centers yesterday was because of a change in a long-running project to increase resilience in its busiest locations.

Many sites went down as a result, including Discord and Shopify. The company says even though these locations, which include Mumbai, Osaka, Singapore, Sydney, and Tokyo, are only 4% of its total network, the outage impacted 50% of total requests. 

Cloudflare says the first outage started at 06:27 UTC on June 21. At 06:58 UTC, the company brought the first data center back online, and by 07:42 UTC, all data centers were online and working correctly.

In a statement on its blog, Cloudflare says, "We are very sorry for this outage. This was our error and not the result of an attack or malicious activity."

The company says over the last 18 months, it has been working on converting all of its busiest locations to a more flexible and resilient architecture. During this time, Cloudflare converted 19 of its data centers to this architecture, internally called Multi-Colo PoP (MCP).

Cloudflare says this new architecture provides significant reliability improvements, allowing it to run maintenance in those locations without disrupting customer traffic. But those locations also carry a substantial proportion of the Cloudflare traffic, so any problem there can have a wide impact.

In order to be reachable on the internet, networks like Cloudflare make use of a protocol called BGP. As part of this protocol, operators define policies that decide which prefixes (a collection of adjacent IP addresses) are advertised to peers (the other networks they connect to) or accepted from peers.

Cloudflare says these policies have individual components, which are evaluated sequentially. The result is that any given prefixes will either be advertised or not advertised. A policy change can mean a previously advertised prefix is no longer advertised, known as being "withdrawn", and those IP addresses will no longer be reachable on the internet.

The company says while deploying a change to its prefix advertisement policies, a re-ordering of terms caused it to withdraw a critical subset of prefixes.

Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. However, the company says it has backup procedures for handling such an event and used them to take control of the affected locations.

But F5 Labs threat research lead David Warburton says this should be a reminder of the dangers of the centralisation of large cloud solution providers.

"In a traditional internet app deployment model, an outage of a server or misconfigured application might take out a single website," he says.

"But similar problems with a cloud solution provider can end up taking out all of their customers, resulting in not one website being taken offline, but hundreds or thousands. The impact can potentially affect organisations' digital experiences, revenues and reputations." 

Warburton says cloud solution providers provide immeasurable benefits to their users, but the re-centralisation of the internet through these cloud solutions is now causing the very problems the original design of the internet was intended to avoid through redundancy. 

ServerChoice commercial director Adam Bradshaw says outages can cause businesses serious reputational and financial damage. 

"For a smaller or medium-sized business, the risk associated with such failures in service can be critical. Even short outages in the digital economy could cause damage," he says.

"Diversifying an IT environment reduces the likelihood of third-party outages negatively impacting a business. Owning the hardware gives an organisation more control, with IT components such as colocation services providing an organisation with a backup when cloud services fail."

Related stories
Top stories
Story image
Tech job moves
Tech job moves - Bitdefender, Cohesity, Fortinet & MODIFI
We round up all job appointments from June 27-30, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Financial results
Margins & revenues up at New Zealand arm of Acer Computer
We look at the local financial statements of Taiwanese manufacturer Acer Computer Inc.
Story image
Robotics
Evonik relies on Getac F110 tablet to control autonomous robot
The aim of the project is to evaluate the practicality of an automated robotic maintenance and inspection solution in the chemical industry.
Story image
Design
Hands-on review: Phillips Hue starter kit
The set-up was one of the simplest I have ever had to do. It is also where the products shine. I was able to connect the bridge, the three lightbulbs and the strip within 10 min.
Story image
Infrastructure
Oracle Cloud Infrastructure expands distributed cloud services
“Distributed cloud is the next evolution of cloud computing, and provides customers with more flexibility and control in how they deploy cloud resources."
Story image
Microsoft
Digital innovation could shape the future of NZ - Microsoft
With cloud technologies available to more people around the world than ever before, it is not only businesses who will benefit from using them.
Story image
Gaming
Hands-on review: 16GB PNY XLR8 Gaming EPIC-X 3600MHz DDR4
PNY sent over its 16GB XLR8 Gaming EPIC-X 3600MHz CL16 DDR4 module kit for testing. The kit contains two 8GB dual-channel modules with RGB lighting.
Story image
Cybersecurity
Zscaler launches co-located data centres in Canberra and Auckland
The investment will offer public and private sector enterprises greater resilience in support of their zero trust cybersecurity posture.
Story image
Finance
Airwallex launches global payment services in New Zealand
The launch will enable businesses in New Zealand to tap into Airwallex's global payments services, offering an alternative to traditional banks.
Story image
Christchurch
Enable launches free Wi-Fi in Christchurch city centre
Fibre broadband provider, Enable, and the Christchurch City Council have launched their new Christchurch Free Wi-Fi service in the central city. 
Story image
Microsoft
ASI Solutions named finalist of Microsoft Surface Partner of the Year
"ASI Solutions has a strong Microsoft focus, building value by helping customers maximise investment in modern workplace solutions."
Story image
Data
MYOB improves data visibility and user access with Snowflake
"Solutions such as Snowflake allow us to better understand our customers and make evidence-based decisions on what features work best for them."
Story image
Salesforce
Data crucial to capture shoppers' wallets post-COVID
First-party data strategies key to driving personalisation, customer satisfaction, and long-lasting relationships according to a new report.
Story image
Mergers and Acquisitions
SAS acquires Kamakura to propel risk technology innovation
Underscoring SAS growth in the domain-specific solutions space, the acquisition will enable SAS to greatly enhance the breadth of its risk solutions portfolio. 
Story image
Cybersecurity
Email threats spike 101%, remains a top attack vector
"Each year we see innovation in the threat landscape, but each year email remains a major threat to organisations."
Story image
Microsoft
TO THE NEW unveils A/NZ Managed Services for Microsoft Azure
TO THE NEW has released Managed Services for Microsoft Azure to meet the growing demand in the A/NZ market and globally.
Story image
Cybersecurity
Tech and data’s role in the changing face of compliance
Accenture's study found that 93% of respondents agree or strongly agree new technologies such as AI and cloud make compliance easier.
Story image
Cybersecurity
Zero trust security adoption rises 27% in just two years
A survey of WAN managers has revealed that multi-factor authentication and single sign-on are the top zero trust features implemented.
Story image
Revenue
Datacom announces revenue of $1.45 billion, fall in profit
Growing market pressures and border closures saw Datacom place increased focus on talent development initiatives for both existing and future employees.
Story image
Identity and Access Management
Ping Identity named a Leader in Access Management
Ping Identity has been named a leader in the 2022 KuppingerCole Leadership Compass report for Access Management. 
Story image
WatchGuard Technologies
Ransomware volume doubled 2021 total by end of Q1 2022
Ransomware detections in the first quarter of this year doubled the total volume reported for 2021, according to a new report. 
Story image
Internet of Things
ManageEngine wins big in IDC MarketScape assessment
ManageEngine's Endpoint Central service has been recognised as a leader by IDC MarketScape in several categories including Internet of Things device deployments and UEM software for SMEs.
Story image
Cybersecurity
FIDO Alliance releases guidelines for optimising UX with FIDO Security Keys
The new guidelines aim to accelerate multi-factor authentication deployment and adoption with FIDO security keys.
Story image
Dicker Data
EXCLUSIVE: Why women in IT makes good business sense - Dicker Data
The Federal government wants to bolster female participation in the tech industry to at least 40% by 2030. Here's how one homegrown Australian company has already reached that goal.
Story image
Cybersecurity
Why is NZ lagging behind the world in cybersecurity?
A recent report by TUANZ has revealed that we are ranked 56th in the world when it comes to cybersecurity - a look into why we're so behind and what needs to be done.
Story image
Hybrid workforce
Why hybrid working is here to stay and how to ace it
Citrix's new report reveals hybrid workers are more productive and engaged at work than their office and completely remote counterparts.
Story image
Entelar
How TruSens air purifiers can create healthier workspaces
The pandemic has heightened our awareness of our own and others’ health, and made us all much more conscious of the environments we work in.
Story image
Hybrid Cloud
Advent One acquires Layer 8 Networks, complements hybrid cloud offering
The acquisition comes at a time of surging demand in hybrid cloud, network virtualisation and network security.
Story image
Cybersecurity
Unknown connections: How safe is public WiFi in Aotearoa?
If it's not your own household WiFi, then who has control of your data and is your connection actually safe?
Story image
Ransomware
Secureworks reveals new information on BRONZE STARLIGHT threat group
New research from Secureworks has uncovered new information on the Chinese threat group BRONZE STARLIGHT and how they are using targeted ransomware to initiate complicated attacks.
Story image
Cyber Criminal
Identity and access: the fight is on
Blue team defenders are used to protecting our data, applications, and users with access controls and other security mechanisms, which is why attacks like this are especially challenging when they target identity and access control systems.
Story image
Cybersecurity
Blasé attitudes to cybersecurity by business a national risk
The largely unregulated state of cybersecurity in NZ, and consequential ambivalence of most businesses, risk hurting the country's trading prospects.
Story image
New Relic
New Relic looks to observability market with new partner programme
New Relic has announced the availability of New Relic Partner Stack, an enhanced programme expanding New Relic’s commitment to partners.
Story image
Great Resignation
New SAP study uncovers impact of 'the great resignation'
Coined in 2021, the phrase 'the great resignation' refers to millions of employees globally leaving their jobs. The phenomenon is real and impacting SMEs.
Story image
MSP
Video: 10 Minute IT Jams - An update from CyberArk
Olly Stimpson joins us today to discuss the importance of MSP programmes and how MSP partners are experiencing success with CyberArk.
Story image
Samsung
Monitors are an excellent incentive for getting employees back
The pandemic has taught us that hybrid working is a lot easier than we would’ve thought, so how can the office be made to feel as comfortable as home? The answer could be staring you in the face right now.
Story image
Citrix
The best ways to attract young talent during labour shortages
New research from Citrix reveals hybrid working and ventures into the metaverse are top of mind for Gen Z workers.
Story image
Web Development
Whitecliffe fosters careers for the future of tech
Do you want a career in Information Technology, Networking, Web Development, Software Development, or are you looking to upskill?
Story image
Cloudian
Cloudian, Vertica to deliver on-premise data warehouse platform
"We’re enabling our customers to capitalise on a leading object storage platform and maximise the value of their digital assets.”
Story image
BitTitan
How to achieve your monthly recurring revenue goals
Monthly recurring revenue (MRR) is the ultimate goal, the most important issue on which anyone in the IT channel should focus.
Story image
Hybrid Cloud
HPE GreenLake advances hybrid cloud experience with new services
"The innovations unveiled today further build on our vision to provide the market with an unmatched platform to spur innovation and drive transformation.”
Story image
Microsoft
SAS wins Microsoft ISV 2022 Partner of the Year award
"We formed the SAS and Microsoft strategic partnership with a shared goal of making it easier for customers to drive better decisions in the cloud."
Story image
Review
Hands-on review – CleanMyMac X utility for macOS
We get hands-on with a useful utility that macOS users never thought they needed but probably do.
Story image
Dark web
Cybercrime in Aotearoa: How does New Zealand law define it?
‘Cybercrime’ is a term we hear all the time, but what exactly is it, and how does New Zealand define it in legal terms?