Comcast can now pinpoint fiber optic cable breaks within minutes

Thanks to the late-night inspiration of engineer Venk Mutalik, Comcast has figured out how to pinpoint breaks in its fiber optic cables throughout its network within 90 seconds.

This enables the company to dispatch repair crews to the fiber breaks immediately, and it gives Comcast some defense against its greatest enemy: construction backhoes that might accidentally sever buried cables.

The ability to restore service is particularly important during the pandemic, when broadband connectivity has been critical for just about everybody. Fixing cables quickly is going to become even more important as the networks carry 10 times more data in the coming years. Comcast plans to eventually upgrade to 10G, which will allow it to deliver 10 gigabits a second both upstream and downstream to the 20-plus million subscriber homes on its network.

“We have a relentless focus on connectivity, as it has never been more important,” Elad Nafshi, senior VP of the Next Generation Access team, said in an interview with VentureBeat. “COVID really highlighted the significance of connectivity. And we’re continuously evolving our network. We see our network as the smartest, fastest, and most reliable. But you can always do better.”

Mutalik is an engineer on the Next Generation Access team, and Comcast has taken his idea and implemented it across the company’s network as a way to keep things running smoothly. Mutalik figured out a way to repurpose previous-generation technology to monitor internet traffic. Comcast can now easily send light signals out through the fiber-optic network, and the light beams hit their destination and bounce back. The engineers can measure how long it takes to do this if the network is normal. The light bounces back more quickly if there is a break in the line, and they use the difference in timing to calculate the precise distance to the break and exactly where it is.

On top of that, they can use the same tests to figure out whether the fiber lines are fraying or otherwise losing their capacity to transmit data. This gives the company insight into the data traffic on its networks but not the actual data we’re sending to each other in messages and such, Nafshi said.

How the network works
The long-haul fiber optic network equipment that connects cities and towns across the United States relies on thousands of highly specialized devices called ROADMs (reconfigurable add-drop multiplexers) that ensure trillions of bytes of information get to their intended destinations. ROADMs contain “optical spectrum analyzer” chips that are essentially ultra-smart prisms, splitting fiber optic laser light into smaller and smaller beams and pointing them reliably at the right destinations, all in milliseconds.

Every year or so, manufacturers release a newer generation of more powerful and higher-resolution optical spectrum analyzer chips, and network operators replace the previous versions of the devices in their networks. The same is true for other components, like the optical time domain reflectometers, optical sonar-like components that use “optical echoes” to pinpoint fiber imperfections.

Mutalik’s solution
That’s what got Mutalik thinking. He knew previous-generation optical spectrum analyzers and other such components were powerful devices designed to manage fiber optic traffic over thousands of miles. What would happen if Comcast repurposed these mature optoelectronic chips to monitor traffic traveling thousands of feet?

“We’re going to have our fiber assets increase 10 times, or something like that, exponentially increasing over time because everything that we need to do is going to be connected by fiber,” Mutalik said. “And so we have a huge amount of fiber assets that go coast to coast. There are a quarter of a million nodes that we have connected to the headends, and all of them are short distances. The question was ‘Do we know what all those fibers are doing?’”

He added, “‘Do we know what wavelengths on those fibers are? How many wavelengths are going down? How many wavelengths are coming up? And if there was an outage of any kind, how do we know the outage is a fiber card versus somebody hitting a telephone pole and then taking the node out?’”

Comcast’s local fiber optic networks in cities, towns, and neighborhoods cover shorter distances than the long-haul networks connecting cities. Mutalik reasoned that optical components such as optical spectrum analyzers, which are designed to perform pinpoint measurements over thousands of miles, would be that much more effective in managing an optical network that powers a single neighborhood.

Pinpointing problems
Mutalik’s team got together and built a prototype for a system called the XMF. By installing these optical spectrum analyzers in the local fiber networks and developing software to leverage the devices’ capabilities in new ways, the team was able to gain unprecedented visibility into performance along every millimeter of those networks.

“The insight was that if you go back and look at the trailing edge of the technology, you see we have three generations faster and more finer-resolution optical spectrum analyzers. But the older-generation chips are available, and they’re perfectly fine for what we need to do in taking what was in the signal path and migrating it to the monitoring path,” Mutalik said. “And because they’re not going through the signaling, we have created these new test points on our network where the light can get monitored automatically, pervasively, continuously, not affecting the signal path.”

Mutalik added, “And then we have optical spectrum analyzers and the time domain reflectometer send a pulse of light. And then we wait for the echo. And then based on the signal processing of the echo, we can determine how long the light travels before getting reflected and coming back.”

Since XMF leverages previous-generation technology for a new purpose, it makes it more cost-effective to install ultra-precise fiber diagnostic tools at many points throughout local fiber networks, creating significantly more visibility for technical teams.

The XMF platform continuously monitors hundreds of thousands of local broadband optical links every minute across the Comcast network, measuring the optical spectrum and testing the length and quality of the fiber links. In practical terms, this means with XMF installed, network engineers can now pinpoint the exact point at which a local fiber network is experiencing an issue and share the precise geolocation with local technicians.

Comcast can now figure out the exact moment when a fiber cut happens. Then it can correlate this to other events, like construction or bad weather. It’s important to know if it is a fiber cut or a power outage or a malfunction.

“Those are very difficult things to diagnose because they require different actions from technicians,” Mutalik said. “If it is a fiber cut, you have an optical technician go. If a node is malfunctioning, you have a radio frequency (RF) technician go. Sometimes you need hours to triangulate and find out what happened.”

And that results in downtime for subscribers.

When installed, XMF reduces the amount of time it takes for a technician to find and isolate a fiber cut from two hours down to one-and-a-half minutes. That helps the company get customers back online faster. It’s also used to discover and repair degraded fiber performance before customers notice. The team uses machine learning to see how the links degrade over time and how maintenance can keep the network going.

“This is a fantastic example of Comcast innovation,” said Nafshi. “And when you take a step back, visibility of what goes on in a network has never been more important. How do we get that same level of visibility from the network? And that’s really a challenge that gave Venk the reason to look at the hundreds of thousands of miles of fiber that represents our access network.”

Rolling it out
The XMF team is now rolling out a handheld version of the XMF platform that technicians can use in the field to identify fiber link issues in seconds and even diagnose and repair those issues on the spot. As the team works to refine the handheld platform, they are beginning to scale the rack-mounted original version more broadly across the Comcast network.

This is a big step toward 10G since it will be important to continue increasing reliability even as the industry accelerates speed and capacity, Nafshi said. Technologies like XMF and Comcast Octave have helped ensure that even as the network reaches new heights of speed and capacity, it will be more secure and reliable than ever for the customers who depend on it.

“In the investments that we’re making in technology and the progression towards 10G, we’re adding not only more speed,” he said. “We also hit key milestones over the last 12 months. Last year, at the height of the pandemic, we announced 1.25-gigabit-per-second symmetrical services, a trial over our distributed access network. In April, we announced the first lab trial of DOCSIS 4.0 FDX, where we demoed 4 gigabit-per-second symmetrical services. So a ton of network innovation goes into future-proofing our network.”

In the meantime, Comcast has to take care of the network. Nafshi said that earlier this week, a truck hit a telephone pole in Houston, and it took out a fiber that served 75 nodes. The system kicked in and pinpointed the cut location to about five kilometers from the hub site. A team was sent out right away and restored the service quickly, shaving hours off the standard length of an outage like that.

The company tries to communicate with construction teams that use backhoes to reduce the risk of fiber cuts. But stuff happens. In London, a crew drilling a hole hit a fiber backbone and took down a part of London’s internet access for three days. I don’t know about you. But if my internet was down for three days, I’d move into a hotel. Venture Beat