Hi new user. Join now and share your ideas,question's...etc.
0 votes
2 views
by (120 points)
image

Facebook insiders claim the tech giant's seven-hour outage that took all of its services offline was exacerbated by employees working from home as staff were locked out of key data centers and messaging services.

The crisis came on a week of cascading disasters for Facebook, as a whistleblower on Tuesday testified before Congress slamming the company's artificial intelligence content algorithms as harmful and divisive. 

The massive outage on Monday was a minor annoyance for casual Facebook users, and a grave concern to the countless small businesses that rely on the platform and its subsidiaries WhatsApp and Instagram as vital channels to communicate with customers.

Facebook is also used to log in to many other apps and services, leading to unexpected issues accessing shopping accounts, dating apps, smart home devices, and a wide range of other services. 

For Facebook employees themselves, the situation was dire. Striking just after 8.30am in California, the outage left employees arriving at the Menlo Park headquarters locked out as the company's access card system went down, according to multiple reports. 

For the roughly three-quarters of Facebook employees working remotely, the company's internal messaging system Workplace was also knocked offline by the outage, leaving a skeleton crew at the company's main Santa Clara data center cut off from assistance as they raced to debug the network servers. 

Teams dispatched to the main data center also reportedly had difficulty accessing secure rooms to revert faulty configurations on key systems. 

Jonathan Zittrain, director of Harvard's Berkman Klein Centre for Internet and Society, said in a tweet: 'Facebook basically locked its keys in its car.'  

Facebook has said that Monday's historic outage was caused by 'configuration changes on the backbone routers that coordinate network traffic between our data centers.'

It is believed that a faulty update to Facebook's Border Gateway Protocol (BGP), which routes traffic between large private networks and the public Internet, left apps and browsers unable to locate the company's services. 

Engineers were rushed to the company's data center in Santa Clara, California (pictured), to reset the servers manually after the company's internal services went down

Engineers were rushed to the company's data center in Santa Clara, California (pictured), to reset the servers manually after the company's internal services went down

A person claiming to be a Facebook employee said on Reddit that high numbers of staff working from home made the problem worse. The account was later deleted

A person claiming to be a Facebook employee said on Reddit that high numbers of staff working from home made the problem worse. The account was later deleted 

Facebook has led the way in allowing employees to work remotely during the pandemic, and one purported insider said on Reddit that the Santa Clara data center had lower staffing due to pandemic restrictions. 

'The nature of the problem meant Facebook would have needed network engineers to physically access their BGP routers - and due to the pandemic, some of the data centers quite possibly don't have an engineer based on site, or someone who could have immediately started to work on the problem,' Kieron Harding, an IT Infrastructure Engineer at GRC International Group, told DailyMail.com.

'One of the reasons why the outage lasted for as long as it did was because the misconfiguration of the BGP also affected Facebook's physical door access systems - which shut down; meaning engineers couldn't get into the buildings, or secure rooms, to start fixing the issues straightaway,' said Harding. 

Harding and other experts have said an inadvertent mistake or sabotage by an insider were both plausible scenarios.

The failures of internal communication tools and other resources that depend on that company's network compounded the initial error, several facebook login account password employees told Reuters. 

'We want to make clear at this time we believe the root cause of this outage was a faulty configuration change,' Facebook said in its blog post. 'We also have no evidence that user data was compromised as a result of this downtime.'

The Facebook outage is the largest ever tracked by web monitoring group Downdetector. 

The tech giant has around 60,000 employees globally and announced in May that it would be operating with a 25 per cent capacity in its offices once it reopened in July.

CEO Mark Zuckerberg announced in June that the company would allow all full-time employees to work from home if their jobs can be done remotely, and has said he expects half the company's to work remotely over the next decade. 

Data center technicians and other who work of physical hardware would be exempt from remote work, Zuckerberg said.   

New York Times reporter Sheera Frenkel said in a tweet: 'Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren't working to access doors.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.

Related questions

0 votes
0 answers 15 views
0 votes
0 answers 7 views
0 votes
0 answers 2 views
asked Nov 2, 2021 by TimMarzano81 (120 points)
0 votes
0 answers 1 view
asked Sep 14, 2021 by ChristenPate (120 points)
0 votes
0 answers 13 views

Categories

110k questions

659 answers

15 comments

296k users

TIPS: Share your content here, easy and free.(Also don't forget to share your post on social site,blogs..etc).More share's means more views and that means higher ranking post.
...