Author Topic: Why We Were Down Yesterday  (Read 225 times)

Online Solar

  • Hero Member
  • *****
  • Posts: 56658
  • Gender: Male
Why We Were Down Yesterday
« on: November 07, 2017, 05:26:15 AM »
We weren't the only one affected yesterday.


HOW A TINY ERROR SHUT OFF THE INTERNET FOR PARTS OF THE US

a misconfiguration at Level 3, an internet backbone company—and enterprise ISP—that underpins other big networks. Network analysts say that the misconfiguration was a routing issue that created a ripple effect, causing problems for companies like Comcast, Spectrum, Verizon, Cox, and RCN across the country.
Level 3, whose acquisition by CenturyLink closed recently, said in a statement to WIRED that it resolved the issue in about 90 minutes. "Our network experienced a service disruption affecting some customers with IP-based services," the company said. "The disruption was caused by a configuration error." Comcast users started reporting internet outages around the time of the Level 3 outages on Monday, but the company said that it was monitoring "an external network issue" and not a problem with its own infrastructure. RCN confirmed that it had some network problems on Monday because of Level 3. The company said it had restored RCN service by rerouting traffic to a different backbone.

https://www.wired.com/story/how-a-tiny-error-shut-off-the-internet-for-parts-of-the-us/
Koolaid is for kids, TEA is for adults

Offline topside

  • Sr. Member
  • ****
  • Posts: 458
  • Gender: Male
  • Work with what is and toward what should be.
Re: Why We Were Down Yesterday
« Reply #1 on: November 07, 2017, 06:13:05 AM »
Remember when the primary tech. was a TV, land-line phone, and power to the home / business? The phone and power were very reliable compared to today's standards.

I went to a presentation by a millennial the other day - very bright, promising young man. He was using an iPad to control his notes and it quit on him. The topic was not part of the discussion, but he blurted out, "Tech only works about half the time."

That comment stuck with me. It does seem like I'm always having to make due with the new tech - computer issues, cell phone issues, marginal service. dropped connectivity. It's interesting how the current society has learned to put up with marginal stability for the extra communications features.  And how we move so much more data but the information you want is buried underneath it. I joke with my younger friends how that cellular device has this amazing feature built in that they may not know of ... they can put in a number and actually talk with someone who is far away from them.  :thumbup:

Online Solar

  • Hero Member
  • *****
  • Posts: 56658
  • Gender: Male
Re: Why We Were Down Yesterday
« Reply #2 on: November 07, 2017, 08:39:38 AM »
Remember when the primary tech. was a TV, land-line phone, and power to the home / business? The phone and power were very reliable compared to today's standards.

I went to a presentation by a millennial the other day - very bright, promising young man. He was using an iPad to control his notes and it quit on him. The topic was not part of the discussion, but he blurted out, "Tech only works about half the time."

That comment stuck with me. It does seem like I'm always having to make due with the new tech - computer issues, cell phone issues, marginal service. dropped connectivity. It's interesting how the current society has learned to put up with marginal stability for the extra communications features.  And how we move so much more data but the information you want is buried underneath it. I joke with my younger friends how that cellular device has this amazing feature built in that they may not know of ... they can put in a number and actually talk with someone who is far away from them.  :thumbup:
Yep, I lost a huge sale years back because my laptop decided it was time to lock up, right in the middle of a presentation.
I had everything ready to go, parts, labor, installation with a follow up agreement for support and a receipt ready to print, and bang, nothing.
This little glitch cost the sale because it made me look incompetent and unprepared, and they were right, I should have printed it out, pics and all.
Problem was, this wealthy family lived extremely rural on 10000 plus acres+, hated tech on all levels, almost like Amish in a sense, so my failed use of tech affirmed their hate of all things electronic, so the sale was lost.
The son was the one that set up the meeting, and he warned me, his parents were very stubborn for change, so I should have known that tech wasn't actually welcome, but I thought I would be able to wow them with this new crap, boy was I wrong, and to date, they still live in the 19th century and only use a generator for the wife to vacuum.

They live a comfortable life, uncomplicated by modern tech and they're quite happy, something a lot of people could learn from.
Koolaid is for kids, TEA is for adults

Offline topside

  • Sr. Member
  • ****
  • Posts: 458
  • Gender: Male
  • Work with what is and toward what should be.
Re: Why We Were Down Yesterday
« Reply #3 on: November 07, 2017, 01:34:46 PM »
Yep, I lost a huge sale years back because my laptop decided it was time to lock up, right in the middle of a presentation.
I had everything ready to go, parts, labor, installation with a follow up agreement for support and a receipt ready to print, and bang, nothing.
This little glitch cost the sale because it made me look incompetent and unprepared, and they were right, I should have printed it out, pics and all.
Problem was, this wealthy family lived extremely rural on 10000 plus acres+, hated tech on all levels, almost like Amish in a sense, so my failed use of tech affirmed their hate of all things electronic, so the sale was lost.
The son was the one that set up the meeting, and he warned me, his parents were very stubborn for change, so I should have known that tech wasn't actually welcome, but I thought I would be able to wow them with this new crap, boy was I wrong, and to date, they still live in the 19th century and only use a generator for the wife to vacuum.

They live a comfortable life, uncomplicated by modern tech and they're quite happy, something a lot of people could learn from.

Nice story.  :thumbup:

Online Solar

  • Hero Member
  • *****
  • Posts: 56658
  • Gender: Male
Re: Why We Were Down Yesterday
« Reply #4 on: November 07, 2017, 01:42:13 PM »
Nice story.  :thumbup:
:lol: :lol: :lol: :lol:
Nah, it sucked. I lost 30 grand on that deal. :sad:
Koolaid is for kids, TEA is for adults

Offline Hoofer

  • Hero Member
  • *****
  • Posts: 4279
  • Gender: Male
  • HAM Radio - the last form of free expression
Re: Why We Were Down Yesterday
« Reply #5 on: November 07, 2017, 03:34:17 PM »
Quote
"Folks are looking to tweak routing policies, and make mistakes," Arbor Networks' Dobbins says. The problem could have come as CenturyLink works to integrate the Level 3 network, or could have stemmed from typical traffic engineering and efficiency work.

This is *probably* what happened.   Reminds me of a know-it-all-kid walking into his dad's business, heads to the Data Center and starts pulling plugs because the patch panel would look "neater" if everything was neatly plugged in "his way".

Another example, a network engineer sees route capacity is getting a little strained, and without checking any circuit design, decides to start redirecting traffic over "unused routes" - which are actually the Diverse Routes, Protect Paths, the secondary stuff that sits practically idle until the Main traffic route goes down.  The discussions that follow are hilarious to some degree.. to everyone - but the customer who is actually PAYING for that protect path.   Right about the moment they insist the path is available ... it's time to bridge the customer onto the call, for a good butt chewing.

The problem is multi-fold, IMO.  A company wants FAST, reliable internet access.  The salesperson sells them an unrestricted "pipe", not shared with anyone else, low latency, -and- a second pipe is added for redundancy/reliability, built over a completely separate path.   The secondary path is taking the l-o-n-g route to the same place, because nobody wants to wait a few more Milli-seconds of Latency, but it's always up, ready and maybe it takes some non-critical, non-time sensitive, traffic, like backing up a server at 2am.   During the day, nothing, no traffic... until there's a fiber cut on the Main path.   Say the same company wants a secondary path, a "back channel" for just inter-office communications, 100% immune to DDoS attacks, it also fits the task perfectly.  The Main path might be 1GBit of bandwidth, the secondary path, 100Megbit - and it could be with a completely different carrier!   People FORGET, poorly document & change jobs... now it's somebody else's problem.   And it's not new, even back in the plain-old-telephone-service days (POTS, land lines), it wasn't uncommon for a trouble ticket, "Survey Customer, they don't know what numbers or how many lines they have".  Still happens today.

What also happens, some really, really, really smart person, in the USA or abroad, thinks they see a way to really save the company money (and get noticed as an innovator), but... sort of light on the skills of deep diving into RECORDS.  They show real initiative and take *action* - resulting in a customer action, RFO "reason-for-outage", and a healthy adjustment to their bill.  What happens to those people?  Years in college, they got a 4.0GPA, and mesmerized by all those colorful blinking lights... not much left to learn, until they start learning the hard way - then they slow down and start asking questions, double-checking, testing, before they do something disastrous.

Of course, it doesn't stop there, since the Internet is really connected all over the place, USA, World and sometimes... to organizations that really lack the talent, staff and *Dedication* to provide top-flight service a really BIG customer would expect.  That starts with a, "Why the hell should we build FIBER all the way out here, when mom-n-pop already has it?  Let's use them!!!"  The fiber is falling off the 20yr old poles, barely buried 18" deep, and the contractors they hire from Hicks-ville, 3 states away, really couldn't give a damn if they pull the wrong fiber jumpers in the Data Center.   "Hell man, it won't hurt to clean all those fibers, we'll find the right one, and we might even improve the service!", not realizing some of those fibers might be carrying Terabits of traffic for your favorite site(s).   Next time there's an outage, I'd like to whisper in their ear, "But.... just think of all that money you were saving!"

Then there's the company that's changed hands so many times, and churned the entire staff almost as many, in their haste to save money, disconnect power to fill-in-the-blank.... whatever is active on there, screw-em, 'cause they're 1 month from bankruptcy.

A sure sign there's gonna be trouble ahead, when someone says, "Those idiots didn't know what the hell they were doing, we're gonna fix that and show them how it's really done!"  Heard that more than once.  One guy was bawling like a baby, begging to keep his job - he works for someone else now, they threw in the towel on a huge grandiose idea, people couldn't afford.  Just didn't have the *time* to listen to experience.   

Mistakes are going to happen, hopefully never repeated - or at least by the same individual.
All animals are created equal; Some just take longer to cook.   Survival is keeping an eye on those around you...

Online walkstall

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 23152
  • Gender: Male
  • WYSIWYG
Re: Why We Were Down Yesterday
« Reply #6 on: November 07, 2017, 07:52:42 PM »
We weren't the only one affected yesterday.


HOW A TINY ERROR SHUT OFF THE INTERNET FOR PARTS OF THE US

a misconfiguration at Level 3, an internet backbone company—and enterprise ISP—that underpins other big networks. Network analysts say that the misconfiguration was a routing issue that created a ripple effect, causing problems for companies like Comcast, Spectrum, Verizon, Cox, and RCN across the country.
Level 3, whose acquisition by CenturyLink closed recently, said in a statement to WIRED that it resolved the issue in about 90 minutes. "Our network experienced a service disruption affecting some customers with IP-based services," the company said. "The disruption was caused by a configuration error." Comcast users started reporting internet outages around the time of the Level 3 outages on Monday, but the company said that it was monitoring "an external network issue" and not a problem with its own infrastructure. RCN confirmed that it had some network problems on Monday because of Level 3. The company said it had restored RCN service by rerouting traffic to a different backbone.

https://www.wired.com/story/how-a-tiny-error-shut-off-the-internet-for-parts-of-the-us/

I only have problems for about 1 min.  The last min I am think.  I was next door helping fix a Honda 3000 generator all that time. 

A lot of gum and varnishes, they had the same gas in it for 6 years.  That and it being Ethanol gas, also pulled the plug and clean it up.  :lol:   Three hours later and it runs great.
A politician thinks of the next election. A statesman, of the next generation.- James Freeman Clarke

Politics is supposed to be the second-oldest profession.  I have come to realize that it bears a very close resemblance to the first.  ~ Ronald Reagan ~

Always remember "Feelings Aren't Facts."

Offline supsalemgr

  • Hero Member
  • *****
  • Posts: 10809
  • Gender: Male
Re: Why We Were Down Yesterday
« Reply #7 on: November 08, 2017, 04:44:52 AM »
Yep, I lost a huge sale years back because my laptop decided it was time to lock up, right in the middle of a presentation.
I had everything ready to go, parts, labor, installation with a follow up agreement for support and a receipt ready to print, and bang, nothing.
This little glitch cost the sale because it made me look incompetent and unprepared, and they were right, I should have printed it out, pics and all.
Problem was, this wealthy family lived extremely rural on 10000 plus acres+, hated tech on all levels, almost like Amish in a sense, so my failed use of tech affirmed their hate of all things electronic, so the sale was lost.
The son was the one that set up the meeting, and he warned me, his parents were very stubborn for change, so I should have known that tech wasn't actually welcome, but I thought I would be able to wow them with this new crap, boy was I wrong, and to date, they still live in the 19th century and only use a generator for the wife to vacuum.

They live a comfortable life, uncomplicated by modern tech and they're quite happy, something a lot of people could learn from.

I was in sales management and the last gig in my career was traveling the country training and mentoring sales reps. Our technology was advanced enough that reps could carry their laptops into a call and type during the visit. I opposed this. I always used an old fashioned Manila folder with the agencies name clearly on it. I never carried my laptop unless there was something specific I could use it for. This was designed to let the agency know they were important and I knew more about their business with my company than they did. This added value. Using the laptop is a distraction to the conversation and eye contact is lost. My colleagues appreciated how it worked, but could not embrace it as the bean counters in charge felt technology was the only way to go. They totally ignore that relationships and knowing what is important to the client is what gets sales. There is where we are today.
"If you can't run with the big dawgs, stay on the porch!"

Online Solar

  • Hero Member
  • *****
  • Posts: 56658
  • Gender: Male
Re: Why We Were Down Yesterday
« Reply #8 on: November 08, 2017, 06:36:37 AM »
I only have problems for about 1 min.  The last min I am think.  I was next door helping fix a Honda 3000 generator all that time. 

A lot of gum and varnishes, they had the same gas in it for 6 years.  That and it being Ethanol gas, also pulled the plug and clean it up.  :lol:   Three hours later and it runs great.
Yeah, I saw it was down, thought it might be on my end, went back to doing some tractor work, came back a few hours later and it was still down.
I called Taxed to tell him, we got distracted by all the good news in the world, how the dim party collapsed, economy improving, left imploding all around the globe, nothing sticks anymore, people are sick to death of Marxists in every form, SA is cleaning house, Podesta's are tied to it, as well as Uranium 1, Clinton Foundation, we were literally laughing at the fact we've been calling this for some time now and it's all coming to fruition, when suddenly the forum came back up. Oh Yeah, the forum... :lol:

So I forgot to have Taxed text the server. Good news'll do that to ya. :thumbsup:
Koolaid is for kids, TEA is for adults

Online Solar

  • Hero Member
  • *****
  • Posts: 56658
  • Gender: Male
Re: Why We Were Down Yesterday
« Reply #9 on: November 08, 2017, 07:01:24 AM »
I was in sales management and the last gig in my career was traveling the country training and mentoring sales reps. Our technology was advanced enough that reps could carry their laptops into a call and type during the visit. I opposed this. I always used an old fashioned Manila folder with the agencies name clearly on it. I never carried my laptop unless there was something specific I could use it for. This was designed to let the agency know they were important and I knew more about their business with my company than they did. This added value. Using the laptop is a distraction to the conversation and eye contact is lost. My colleagues appreciated how it worked, but could not embrace it as the bean counters in charge felt technology was the only way to go. They totally ignore that relationships and knowing what is important to the client is what gets sales. There is where we are today.
True wisdom SSM.  :thumbup:
I hope people reading take this to heart. Not being tech-heavy shows preparedness, a folder with the company's name on it and a briefcase full of prepped data will exude confidence, as opposed to some guy saying, "Hang on, I'll text so and so for an answer.

It's like when we were kids, wearing a suit and tie was square, it meant you were bending to the man, "anyone over 30" was an enemy (Marxist message creating division), but who got the job? The kid that took the time to impress, threw off the leftist bull shit being pushed on the youth (hippy generation).

No, showing confidence in oneself, not following PC nonsense with crap like asexual attire, if you're a man, wear a damn suit, if you're female, wear a nice skirt for Christs sake, or antisocial behavior like pants off your ass, tats is a job killer as well, and one more hurdle you've created for yourself, especially if you can't hide them.

My advice to young people wanting to get hired. Study the history of the business of intertest, know it's strengths and weaknesses, praise the good it does, and if given an opportunity, explain what you have to offer long term, assuming you're looking for a career, or ladder to a better company.
The last thing you want to come off as, is one of the ten million drones claiming they wear metal and tatts to be different, just like 99% of the losers that will be stuck at a lowly retail job, or burger flipper until tech replaces their worthless asses and no one will hire them.

Funny, I spelled tatttts wrong when my finger stuck to the t, so spellcheck comes back and says "Target" is the correct spelling.
Boy how prophetic that is, a target most people avoid.

Tatts and piercings scream follower. Someone not bright enough to avoid antisocial trends being pushed by Marxists trying to destroy a cohesive culture.
Koolaid is for kids, TEA is for adults

Offline supsalemgr

  • Hero Member
  • *****
  • Posts: 10809
  • Gender: Male
Re: Why We Were Down Yesterday
« Reply #10 on: November 08, 2017, 09:27:41 AM »
True wisdom SSM.  :thumbup:
I hope people reading take this to heart. Not being tech-heavy shows preparedness, a folder with the company's name on it and a briefcase full of prepped data will exude confidence, as opposed to some guy saying, "Hang on, I'll text so and so for an answer.

It's like when we were kids, wearing a suit and tie was square, it meant you were bending to the man, "anyone over 30" was an enemy (Marxist message creating division), but who got the job? The kid that took the time to impress, threw off the leftist bull shit being pushed on the youth (hippy generation).

No, showing confidence in oneself, not following PC nonsense with crap like asexual attire, if you're a man, wear a damn suit, if you're female, wear a nice skirt for Christs sake, or antisocial behavior like pants off your ass, tats is a job killer as well, and one more hurdle you've created for yourself, especially if you can't hide them.

My advice to young people wanting to get hired. Study the history of the business of intertest, know it's strengths and weaknesses, praise the good it does, and if given an opportunity, explain what you have to offer long term, assuming you're looking for a career, or ladder to a better company.
The last thing you want to come off as, is one of the ten million drones claiming they wear metal and tatts to be different, just like 99% of the losers that will be stuck at a lowly retail job, or burger flipper until tech replaces their worthless asses and no one will hire them.

Funny, I spelled tatttts wrong when my finger stuck to the t, so spellcheck comes back and says "Target" is the correct spelling.
Boy how prophetic that is, a target most people avoid.

Tatts and piercings scream follower. Someone not bright enough to avoid antisocial trends being pushed by Marxists trying to destroy a cohesive culture.

Wearing a tie was another technique I used until the last day I worked. Like in most industries "business casual" attire was acceptable in my industry. However, I never embraced it. I felt my wearing business attire set another level of importance to the meeting. I made this part of my presentations in training sessions. I was surprised the president of the company I worked with after I retired, in his forties, bought into my philosophy.
"If you can't run with the big dawgs, stay on the porch!"

Online Solar

  • Hero Member
  • *****
  • Posts: 56658
  • Gender: Male
Re: Why We Were Down Yesterday
« Reply #11 on: November 08, 2017, 09:57:05 AM »
Wearing a tie was another technique I used until the last day I worked. Like in most industries "business casual" attire was acceptable in my industry. However, I never embraced it. I felt my wearing business attire set another level of importance to the meeting. I made this part of my presentations in training sessions. I was surprised the president of the company I worked with after I retired, in his forties, bought into my philosophy.
:thumbsup:
Smart man. There is nothing like tried and true Conservative teachings/values over the millennia, youth may rebel, but once they see results first hand, they readily accept it.
Koolaid is for kids, TEA is for adults

Online walkstall

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 23152
  • Gender: Male
  • WYSIWYG
Re: Why We Were Down Yesterday
« Reply #12 on: November 08, 2017, 10:53:58 AM »
I always used a  folder and a briefcase full of prepped data.  That way I could hand it out if ask for.  I used MSPP when training workers only not management.
A politician thinks of the next election. A statesman, of the next generation.- James Freeman Clarke

Politics is supposed to be the second-oldest profession.  I have come to realize that it bears a very close resemblance to the first.  ~ Ronald Reagan ~

Always remember "Feelings Aren't Facts."

 

Powered by EzPortal