There is no word on when the independent investigation into the XT Network will be complete as Telecom customers suffered a fifth outage on the network.
This time it was customers north of the Bombay Hills who suffered intermittent service, when cellsites connected to the original Radio Network Controller (RNC) in Auckland went down. Telecom hasn’t explained the cause of the outage, but a spokesperson says it isn’t due to migration of cellsites to a second RNC covering the area.
In an email to business clients Gen-i CEO Chris Quin said the issue affected up to 200 cell sites and it was not related to the performance of the Auckland RNCs. “The issue is believed to be due to a variable level of service performance from the network that supports cell sites attached to our Auckland RNCs.”
He claimed it was not an outage, “it was reduced accessibility of mobile services on part of the network for a short period of time.”
There are 470,000 customers on the XT network, with one major customer, AA CIO Doug Wilson labelling today’s issues “bloody disappointing.” He added: “I wish they would keep their website up to date when they have issues like this. It is difficult to let our users know what is happening.”
However, as the AA has retained a mobile data service with a rival network, Wilson says it was only voice that was affected. AA was one company bought over by Gen-i from rivals Vodafone during the push for XT. They had originally planned to put all their mobile business with XT.
Another company was Fonterra, which made the announcement recently that it would be stalling a complete migration to XT until it understood what has caused the outages. A spokesperson told TR today that Fonterra has not resumed its migration.
As Analysys Mason prepares their report on the causes of the XT failures, here’s what Telecom and Alcatel Lucent, the company that built the network, have said publically about the previous outages.
While the cause of the first four service disruptions was different, each one affected the Christchurch RNC which routes XT traffic south of Taupo. Telecom has been clear the outages were not due to overload on the original RNCs, claiming peak traffic has never exceeded 50%.
First outage – 14 December
The outage began at 4am and service wasn’t fully restored until 5pm. Alcatel Lucent chief technology officer Martin Sharrock describes the cause of this outage as an operational problem. That is, a change was made to the network and when the network was brought back up again there was an outage. He says that changes are continually made to upgrade the network to accommodate massive shifts in the population. As an example he told the TUANZ After 5s that at the end of the university term they have to reconfigure the network to take account of the fact that students leave the city and go home for the holidays, “The night they go home you’ve got to go in and change the whole network and the way it works”.
This outage raised alarm bells amongst the business community, with TUANZ chair Chris O’Connell claiming at the time that people rely on their mobile network and “they feel like their throat’s been cut.”
Second outage – 27 - 30 January
This outage was due to a failure in the routing equipment responsible for thousands of connections and channels. It was to take three days before service was fully restored, with customers in Dunedin and the lower South Island most affected. The major issue here was signalling storms – that is when thousands of users try to reconnect at same time, overloading the network and causing, in this case, massive damage.
Sharrock described the cause of this outage as a “strange signalling storm, an unprecedented situation”. Alcatel Lucent experts were flown in from all over the world to help solve the problem. Telecom held a press conference and provided a compensation package worth a total of $5 million to affected customers.
Third outage – 17 February
Disruption to service mainly for customers in the lower North Island as Alcatel Lucent migrated cellsites to a new RNC. This was considered an “operational problem” similar to what occurred during the first outage.
Fourth outage – 22 February
Service to customers south of Taupo was disrupted at around 4pm and was restored early the next morning. Sharrock claims this was a “completely different phenomenon”. It caused Telecom massive embarrassment and the company was forced to shell out for another compensation package in an effort to retain its customers – this time costing the company $10 million.
97% coverage issues
Meanwhile the company has revealed its spending $130 million on upgrading the XT Network, much of it to make good on its promise of 97% network population coverage, which it has admitted it has not achieved.
By the end of June it will have installed a further 47 cell sites and 423 Tower Mounted Amplifiers onto existing sites - all to get the same coverage as its old CDMA network, which is due to be mothballed in 2012.