This too shall pass. Yes, we’ve seen this before: OFDMA + DBC = BSoD

Yes, you have experienced it. Either as a subscriber or technician. The cable modem is on but there is no data flowing through it. Reboot the modem and it works again. It’s the equivalent of the Microsoft Windows blue screen of death (BSoD) and it’s back with a vengeance for cable operators running DOCSIS 3.1 OFDMA with a noisy upstream. What gives? I will cover what and why in this article and recommendations on how to avoid BSoD on OFDMA in the upstream.

What is BSoD and how does it relate to modems?

According to Wikapedia, BSoD, is “A stop error or exception error, commonly called the blue screen of death (BSoD) or blue screen, is an error screen displayed on Windows computers following a fatal system error. It indicates a system crash, in which the operating system has reached a condition where it can no longer operate safely.” ^[1]

For years I and other fellow DOCSIS troubleshooting experts have referred to crashing modems as BSoD’d modems because just like Microsoft Windows it requires a reboot to fix. A crashed modem happens when its internal operating system crashes. Typically, we find that the cable modem stops passing traffic, however, the DOCSIS chipset is still online and shows “online” from the perspective of the CMTS. This creates a unique problem as most monitoring systems query the CMTS directly for the modem online status. This means the CMTS will report the modem as “online,” but if you ask the subscriber, they will respond with “my service is down and after this call I’m going on Facebook to trash talk you [cable operator] since clearly you [cable operator] cannot even tell that my service is down.” Now there are tools you can use to help avoid this situation which I will cover below.

Crashed modems can be hard to detect and they may result in a call to a CSR from the subscriber. The CSR usually will instruct the subscriber to reboot the modem. To some subscribers this will cause just a bit of an annoyance, however, to other subscribers this will cause fear, panic and irritation. Being the resident helpdesk for my friends, family and neighbors I can say this with some degree of confidence that all these emotions are felt by subscribers.

Crashed modems are not new to the industry. It happened when DOCSIS 2.0 modems got off the ground, then again when DOCSIS 3.0 modems rolled out. With new technology there will always be bugs to work out of the system. Along the way vendors fixed the bugs, just like Microsoft has fixed their bugs. We rarely see the dreaded blue screen of death in Windows anymore and it is infrequent in cable modems as long as their firmware is kept up to date. Unfortunately, but to be expected, crashed cable modems are back again for DOCSIS 3.1 when they are locked to an OFDMA channel with lots of noise. But why is this noise causing the modems to crash?

Upstream OFDMA — OUDP and DBC

To understand why a cable modem crashes, we must understand what mechanisms are at play that can cause the potential crash or BSoD. These mechanisms are usually associated with dynamic changes in the modem. This is what often triggered DOCSIS 2.0 and 3.0 crashes, so it’s very likely this is what is causing DOCSIS 3.1 crashes.

When a DOCSIS 3.1 modem first comes online and communicates with the CMTS using an upstream OFDMA channel, the CMTS will first set the modem’s OFDMA upstream data profile (OUDP) to IUC 13. This should be configured to be a low order modulation like 16-QAM so that it is resilient to any noise and ensures the modem will come online. Once the modem is online, the CMTS will check if the upstream supports a higher order modulation for each modem. If so, the CMTS will use a dynamic bonding change (DBC) messages to move each modem a higher order modulation profile. This change is done dynamically.

Dynamic changes to modems have historically caused them to crash. I have seen this on DOCSIS 2.0 modems when we started load balancing and for dynamic quality of service (DQoS), and DOCSIS 3.0 modems for channel bonding changes, load balancing and DQoS. I’m confident that the root cause we are seeing with DOCSIS 3.1 modems when using OFDMA on noisy upstreams is likely related to DBC messages. These messages are the direct result of return path noise causing frequent profile change requests from the CMTS. According to a colleague, this can occur up to every five minutes if configured in the CMTS. An error in the modem’s code eventually causes it to crash, but the DOCSIS chipset remains online, confounding our monitoring systems and frustrating our subscribers.

So why do OFDMA if there are problems?

I have authored articles on OFDMA, such as “OFDM and OFDMA” https://broadbandlibrary.com/ofdma/) and covered this topic several times on my livestream and podcast so I will just briefly highlight why OFDMA here. One reason is that OFDMA gives the cable operator more bits per second per hertz in the upstream than single carrier QAM (SC-QAM) channels. A second great reason is that OFDMA also has a much better error correction algorithm called low density parity check, compared to SC-QAM’s Reed Solomon’s error correction. For these two reasons alone, OFDMA is a great technology that we get with DOCSIS 3.1 to start using in the upstream to meet the traffic demands of our subscribers. Third, it allows you to take advantage of the coax out there and get more ROI on an existing plant. Cost of deployment cannot be overlooked. However, when deploying OFDMA we should follow some simple recommendations, which I will outline next.

Recommendations

The technology of OFDMA and the DOCSIS 3.1 specification is sound. It is just rolling out so early adopters will experience the most pain and reward. Relaying the issues back to vendors is especially important so that they can more rapidly resolve the errors in the firmware. It will require some patience before vendors resolve the bugs and issue firmware updates to fix the problems many operators are experiencing.

In the meantime, while we wait, here are some recommendations:

• Place OFDMA channels above 42 MHz: For those fortunate with 5-65 MHz or 5-85 MHz return paths, keeping four 6.4 MHz SC-QAM signals below 42 MHz and then loading one OFDMA channel above 42 MHz is ideal. Your SC-QAM channels should be fully optimized with dynamic profile modulation up to 64-QAM. Then, by placing your OFDMA channel above 42 MHz, you’re ensuring its likely to be in a low noise environment. A bonus is that subscribers with in-home amplifiers will not run into partial-mode issues and your OFDMA channel may even be able to run up to 1024-QAM. For now, please keep your OFDMA channel far away from low frequency noise as shown in Figure 1.
• Place your OFDMA channel above 30 MHz: So, you don’t have an 85 MHz return. No problem, ATDMA SC-QAM carriers running a modified modulation profile using dynamic interleaving, long preambles and dynamic modulation profiles do amazingly well at lower frequencies with ingress and noise. If you didn’t understand most of the stuff I just said, that’s okay. There is a DOCSIS guy working at your company who does. The important thing is that now you can place your OFDMA channel above 30 MHz and take advantage of that blazing fast upstream speed [some] of your subscribers have been demanding.
• Tools: Tools such as proactive network maintenance (PNM) and profile management application (PMA) can be very helpful. These applications will give you insight into RxMER and can also help optimize upstream OFDMA profiles and downstream OFDM profiles resulting in increased data throughput.
• Deployment: Being on the cutting edge is great. There is value in the return that you see. However, sometimes being on the cutting edge has drawbacks too. I’m working with enough cable operators who are in a painful spot because they chose to be on the cutting edge. I know the vendors are working hard to fix the issues and will stabilize OFDMA very soon. Patience is a virtue.

Continue reading at Broadband Library