Getting the Basics of Dedicated Outbound Troubleshooting

When you troubleshoot a dedicated circuit, you do it in the reverse order that you troubleshoot switched phone lines. If you have a problem calling Hoboken, New Jersey, on a switched phone line, for example, you should perform your troubleshooting before you call your carrier and open up a trouble ticket. When it comes to dedicated circuits, however, you begin your troubleshooting after you open the ticket.

The reason you should open your trouble ticket forthwith is because you don’t have many testing options on a circuit without the aid of a technician from your carrier. You might know that your circuit isn’t working, but only your carrier can tell that 10 of your channels are in an RMB state and 14 are in IDL. Even if you know the problem you are experiencing is caused by your hardware, you should still open a trouble ticket to receive help validating the source.

 Remember  The troubleshooting steps in this chapter deal with a simple dedicated long-distance voice circuit with one local loop provider. If you have a type 2 or type 3 circuit that involves more than one carrier to deliver the local loop, the troubleshooting process is more involved and beyond the scope of this book. One thing I do suggest is that you put all the carriers involved in your circuit on a conference call at the same time.

Step 1: Rebooting your hardware

If your circuit is down hard, either in an RMB or CFL state, it’s always good to begin your troubleshooting by rebooting your multiplexer and/or CSU.

 Tip  The two pieces of hardware that are actually talking to each other on the circuit are your multiplexer and the computer card at your carrier that functions like a multiplexer, so rebooting the hardware on your side may resolve the issue. If rebooting brings your circuit back up, you should still follow up with the trouble ticket to determine the root of the problem. Most problems don’t happen once and then disappear forever. If troubleshooting indicates a hardware problem, you ought to replace the faulty hardware while it’s still limping along, instead of waiting for it to fail completely.

Understanding your trouble ticket options

 Remember  You can open special trouble tickets for specific issues. Not only can you open a ticket for repair, you an also open an information-only ticket, or a tech assist ticket. Opening an information-only ticket allows you to officially relay information to your carrier. If you want to tell your carrier that you will be disconnecting your hardware for the weekend because you are moving your phone closet to another suite, for example, the informational trouble ticket relays the information and gives you quicker access to a technician if you need one.

Use a tech assist ticket to schedule nonemergency testing between your carrier and your hardware vendor. If you are replacing your hardware, for instance, you should coordinate with your carrier to be either on the phone with your hardware vendor while making the change, or at least to be immediately available to the vendor.

The last thing you need when you encounter a problem during such a change is to wait hours to speak to a technician. You might not be inconvenienced if the maintenance takes place after normal business hours, but your wallet will still be skinnier after you have to pay for your vendor to sit around and wait for the call.

Rebooting your hardware isn’t helpful for troubleshooting call quality issues, because they will generally be the result of something being set up wrong or a progressive failure in a section of your hardware. The first problem could be anything from a protocol mismatch to a configuration mismatch.

Here are some important rebooting do’s and don’ts:

• Do make sure more than one person knows how to reboot the phone hardware. You and someone else in your office (your backup) should know how to reboot every piece of your phone system. It’s a waste of time and money to call out your hardware vendor to push one button. If you don’t know how to reboot your hardware, ask your hardware vendor to show you. If it’s a complicated procedure, have the vendor write it in black pen on the wall of the phone room next to your hardware. It’s well worth having graffiti on your wall to be able to confidently push a button without worrying what will happen (besides, graffiti art is cool).

•  Warning!  Don’t be in the dark about whether your phone system will restore to default settings. It’s uncommon, but not unheard of, for hardware to restore to default settings after being rebooted. When you discuss recy-cling the power of your PBX, multiplexer, and CSU (as well as any other servers or devices attached to these three pieces of hardware) with your vendor, be sure to ask about possible side effects of rebooting. While you are at it, also ask about fixing these side effects if they come up. The solution might be very simple, or it might be, “don’t touch anything; just call me.” Either way, this information is good to know.

• Do make note of the frequency of your reboots. If your circuit drops on a regular basis, be it once a month or once every three months, but immediately comes back to life when you reboot your hardware, your hardware might have a timing issue. A timing issue can be caused by your hardware utilizing a clock for the point of reference on your TDM circuit that doesn’t come from your carrier, or the section of your hardware that handles the clocking on the circuit may be failing. In either case, timing issues cause the phone system to accumulate small slips within the circuit that eventually cause the circuit to fail. You might ignore the problem because your circuit comes back to life as soon as you reboot, but the issue isn’t solved. It will progress over time, and the time it takes for your circuit to fail will gradually become shorter. If this scenario sounds familiar, contact your hardware vendor to check your phone equipment. If this doesn’t make a lot of sense to you, read up on the clock source requirements of dedicated circuits in Chapter 8.

Step 2: Intrusively testing: Looping the CSU

Your carrier can intrusively test a circuit, which renders the circuit useless to you during the duration of the test. Every channel of the circuit being tested must be engaged so you won’t have any room left to either make or receive calls.

 Remember  As soon as your carrier begins intrusive testing on your circuit, all the active calls on your circuit will be disconnected. The testing environment usurps all the bandwidth. Your hardware can’t grab any DS-0 to make or receive a call during an intrusive test. If your circuit is already down, you should release it for intrusive testing. If the circuit is bouncing, or only slightly impaired, you need to make a judgment call. Can your business stand to have the circuit down for 30 to 60 minutes now, or should you wait until after your office closes to release it for intrusive testing?

 Tip  You don’t have to release your circuit for intrusive testing at your carrier’s discretion when you open your trouble ticket. If your circuit is bouncing, ask the carrier to do some preliminary work first (such as accessing the system’s performance monitors). You can always release the circuit for testing after hours, say at 7 p.m. central standard time. This way, you can get basic information, and when you get in tomorrow morning, you can get more information about what the carrier found during the intrusive test.

An intrusive test involves looping the CSU. Looping is a general term used in telecom that describes the act of sending an electrical signal to a specific piece of hardware where it’s bounced back to your carrier. Your carrier technicians actually send a stream of data down the circuit, possibly consisting of all 1s or all 0s. If every “1” the technician sends in a test effectively hits the desired endpoint and comes back to your carrier, your circuit has solid continuity from your carrier to that point.

If the idea of looping doesn’t make sense to you, imagine it another way. If you have a garden hose with leaks, the easiest way to find them is by placing a spray nozzle on the end of the hose and closing it so no water can escape. After the end is sealed off, you can check for leaks by turning on the faucet and looking for any water springing out of it. Intrusive testing does the same thing, but with data. If you send a series of 27 consecutive 1s down the circuit and only 25 return, you know that a pair of your 1s were lost somewhere between the point at which the data originated and the looped piece of hardware that is sending it back to you. Losing data means that you have a problem in that section of your circuit.

 Technical Stuff  There are many tests that your carrier can use when looping a piece of hardware. They have names like QUAZI, 4 and 8, 6 and 2, and my personal favorite, all 1s. Some tests are more rigorous on a circuit than others, so have your carrier try a few of them if you don’t feel confident with the results of the first test.

Intrusively testing a dedicated circuit follows a very methodical sequence. Your carrier’s first test is attempting to loop your CSU. If you don’t know what a CSU is, please check out Chapter 4 for info on your hardware requirements for a dedicated circuit. If you are more concerned with how the CSU is positioned in the circuit, check out Figure 13-1 to see where it lives in the dedicated circuit landscape.

Figure 13-1: The CSU is the first piece of hardware your carrier intrusively tests.

Dealing with synch issues caused by Dialogic cards

 Remember  If you are using Dialogic cards as the multiplexing interface on your side of the circuit, you may have problems when you reboot your hardware. There is a known issue that exists between Dialogic cards and some switches that the carriers use. If you find that your Dialogic cards don’t automatically synch up when you reboot, or if individual channels lock up when you dial to a bad phone number (where you receive a recording or a fast busy signal), you need to enable the CRC-6 setting on your Dialogic card to resolve the issue. You might also need to reboot your Dialogic card before the programming change takes effect. After the setting is enabled, your channels shouldn’t fall out of sync with your carrier any more.

Figure 13-1 shows that the CSU is the first piece of hardware your carrier identifies as your responsibility. There is a cable from the NIU to your CSU that is also your responsibility, but it isn’t generally loopable, so the CSU is the default piece of hardware used as the first point for testing.

 Remember  Not all CSUs are identical. Your CSU might not be a free-standing piece of hardware as portrayed in Figure 13-1, and might instead be an integrated card housed within your multiplexer. It’s uncommon, but not unheard of, for a CSU to not be loopable. This isn’t a huge stumbling block, but you simply need to be aware of this bit of trivia. Your carrier might try to imply that your hardware is defective, because the carrier can’t loop your CSU.

If looping the CSU fails

Here are the possible meanings of a failure to loop the CSU:

• If you can’t loop the CSU, this is a strong indication that there are problems somewhere in your local loop.

• If you are testing to identify the source of a circuit failure issue, and your carrier can’t see the CSU to loop it, there’s a break in the circuit somewhere in the area being tested.

• If you are troubleshooting for a call quality issue and you can loop the CSU (indicating that you have electrical continuity) but you receive errors, some piece of hardware within the testing area is probably dying a slow death.

 Remember  These tests don’t indicate that your CSU is the problem; they simply indicate that something between your CSU and the CFA point is the problem. The only way to isolate the issue is to proceed to “Step 3: Looping the NIU.”

If looping the CSU is successful

If your carrier can loop your CSU without a problem (for circuit failure issues) and can run test patterns to it without taking any errors (for circuit quality issues), the entire section of the circuit you are testing is clean. Looping your CSU generally causes the equipment on the other side of it, usually your multiplexer, to idle up all the channels.

Looping the CSU not only bounces the circuit signal back to your carrier, but also bounces the signal that it’s sending back to your multiplexer. If your MUX doesn’t idle up to the back of the looped CSU, your next course of action is to have your hardware vendor test your MUX to determine whether it’s the point of failure.

 Remember  Looping a piece of hardware doesn’t confirm that everything is wired correctly to the device being tested; it simply confirms that you have electrical continuity. See the following section to make sure your wires aren’t crossed.

Making sure your crossover cables aren’t straight-through cables

Dedicated circuits have designated wires to transmit and receive signals. Even if these transmit and receive wires are crossed between the NIU and the CSU, you can still loop the CSU; but in that case, the T-1 level of your circuit won’t idle up. If this situation occurs, you need to check the configuration of the cable you are using to connect your NIU to your CSU.

There are two main varieties of cables in telecom, straight-through cables, and crossover cables. The wires that make up cables are numbered for easy reference, allowing you to understand why they have their specific label. Figure 13-2 shows a four-wire circuit, with each wire identified by a number. You can see how the black wire #1 on the straight-through cable remains wire #1 at the far end, whereas it ends up as wire #4 on the far end of the crossover cable. Needless to say, it’s very easy to wire a cable incorrectly and end up using the wrong wires for transmit and receive.

Figure 13-2: A comparison of straight-through and crossover cables.

If the CSU can be looped and you have checked the cabling, but you still don’t have dial tone, you need to confirm the configuration of your circuit. In spite of the fact that you may have confirmed the line coding and framing is set for B8ZS/ESF on both ends of the loop, that still doesn’t prove that a piece of hardware isn’t programmed incorrectly. Your carrier can validate that the line is B8ZS/ESF by sending all 0s down the circuit. If the zeros come back without errors, you can rest assured that all the hardware it’s hitting is configured for B8ZS/ESF. Your carrier can run the same type of test for AMI/SF by sending down all 1s. When you’ve validated the line coding, you should confirm the outpulse signal and start (E&M Wink or Immediate, loopstart, or groundstart) on both your hardware and the carrier. If you want more information about dedicated circuit line configuration options, like B8ZS/ESF, see Chapter 8.

 Remember  If the CSU can be looped, and the configuration has been confirmed, the next variable to examine is the hardware on both ends of the circuit. Just as you have a multiplexer that breaks down your T-1 into 24 individual channels, your carrier has a card in its phone switch that does the same thing.

It’s very unlikely that a T-1 level card in your carrier’s switch will fail, but it does happen. If you have more than one circuit and only one of the circuits has been acting up, ask your carrier to reassign one of the known bad circuits to the card that is working on the good circuit. If your hardware instantly comes to life when you are on the known good card, the other card is obviously bad (that was the only variable you changed). If you make the change and your circuit is still down, your hardware is probably the source of the trouble. Move on to the following section.

Using a T-1 test set

If all the previous tests don’t reveal the source of your problem, your next step is to replace your hardware and see whether that brings the circuit up. If this doesn’t resolve your issue, you need to call out a hardware technician with a portable T-1 test set, possibly a Phoenix test set model 5575 or a T-Berd test set made by TTC.

 Tip  If you have a complex phone system, or if your business is in any way based on providing phone service, whoever is servicing your phone system should have a piece of testing hardware. Test sets look like a cross between a tool-box and a 1930s lunch box. They range in price from a few hundred dollars for a used Phoenix 5575A, to several thousand dollars for a fully fleshed out TTC T-Berd or TTC FireBerd with all the available ISDN options.

Having a test set for your dedicated circuit is very helpful, so chat with your hardware provider and see what is the best test set for your application and budget. A refurbished Phoenix 5575A might be all you will need and is well worth the $200 you might end up spending for it on eBay.

 Tip  If you don’t want to buy your own test set, you can always have a hardware technician come out and do the work for you. Call your hardware vendor and set up an appointment as soon as possible.

With a test set, you can run the continuity tests that your carrier typically performs. This allows you to begin testing as soon as you have a problem on your circuit, without waiting for a technician to call you back. After you have a technician on the line testing, you can validate his or her findings, and provide the technician with much greater information on your circuit. You can plug the test set into your circuit between the NIU and your CSU for the greatest vantage point. By placing the test set in your circuit at this location, you bypass your entire phone system. This is a tremendous benefit because the test set acts as an insulator between your hardware and your carrier. If your test set is monitoring the signal coming in from your carrier and it shows that the carrier is sending B8ZS/ESF signaling rather than AMI/SF, there is no way the problem can be related to your multiplexer. All the hardware behind your test set is hidden by it and invisible to your carrier. The minor investment will pay off, because it will no longer be possible for your hardware vendor and your carrier to point fingers at each other. The test set is an independent piece of hardware that you can use to replace your multiplexer in the circuit, and can allow you to dial out on an individual channel if your test set has the correct options. If your problem clears when your carrier is connected to your test set rather than to your multiplexer, your hardware is at fault and your next course of action is a detailed check of your multiplexer and the network behind it.

Step 3: Looping the NIU

If your carrier can’t loop your CSU, the next step to attempt is to drop back one step closer to the CFA point of your circuit and loop your NIU (or one step closer to Nirvana, depending on your perspective). The NIU is the last piece of hardware that your local carrier installed, as shown in Figure 13-3.

Figure 13-3: Dedicated circuit testing: looping the NIU.

By looping the NIU, your carrier can test to the back end of the NIU where your local carrier wires into it. Your goal when methodically validating smaller sections of the circuit from the CSU back to your carrier is to find the section that is without errors or problems. Stepping back your continuity tests (for circuit failure issues) or error tests for a clean span (for quality issues) to a section without issues gives you a starting point on the circuit that you know is good. After you find that good section, you can focus your efforts on the next variable, working your way closer to your CSU, which is the hardware or the span of the circuit that’s most likely the source of the issue.

If you can loop the NIU

If your carrier can loop the NIU but not the CSU, the problem lies somewhere between the back end of the NIU and the front end of the CSU. The cabling or the CSU has a break in the wiring (if your carrier is seeing a lack of continuity on the line, often referred to as an open) or is shorting out (if the carrier sees errors).

 Remember  If you test the CSU and it can be looped, you need to ask whomever provided the inside wiring to repair the trouble. If your local carrier pulled the inside wiring, it might be quicker to have your hardware vendor validate or rerun the cabling, instead of waiting the one to five days to have your local carrier dispatch someone to your site. To test the inside wiring, proceed to “Step 4: Looping to your T-1 jack.”

Getting the scoop on loops

 Technical Stuff  There are two varieties of testing loops in the world of telecom: soft loops and hard loops. CSUs and NIUs generate soft loops, which use software to bounce the signaling on a circuit back to its point of origin. You create hard loops by physically wiring together the transmit and receive lines on a circuit. You typically place a hard loop in a circuit by using a loopback plug, but both loops perform the same function.

If you can’t loop the NIU

If you can’t loop the NIU, there is a problem in the local loop, or possibly within your long-distance carrier’s network. The only way to isolate the issue is to continue working back on the circuit towards the CFA point until you find a good section that doesn’t have errors. In this case, proceed straight to “Step 5: Looping the CFA point” for additional troubleshooting.

Step 4: Looping to your T-1 jack

It’s unlikely that your CSU is connected directly into your NIU. These two pieces of hardware might be in different sections of your building, connected by the inside wiring that ends next to your phone system in a small T-1 phone jack (called an RJ-45 jack). Figure 13-4 shows how the RJ-45 jack sits between your NIU and your CSU at the end of your inside wiring.

Figure 13-4: Inside wiring, including the T-1 jack.

In order to test to the RJ-45 jack for continuity and errors, you have to plug into it with a test set, or use a loopback plug, which is a much cheaper alternative. Loopback plugs are available in both male and female versions so that you can test your cables and the T-1 jack on the wall.

 Tip  The appendix at the end of this book offers detailed descriptions of the loop-back plugs. They are well worth the 50 cents of hardware it takes to make them (although the crimper tool used to clamp down the pins may cost up to $50).

When you place the loopback plug in the jack, you allow your carrier to continue to send signals through the NIU to the phone jack. If the carrier can see the loopback plug and run test patterns without losing any data or receiving errors, you know that the section of the cabling to that point is solid. If your carrier can’t see the loopback plug, or is taking errors to it, but can loop the NIU without a problem, your trouble lives in the wiring between the NIU and that T-1 jack. Figure 13-5 illustrates the portion of the circuit you have to test next.

Figure 13-5: Dedicated circuit testing: looping the T-1 jack.

 Tip  You can extend the test to the end of the cable that feeds into your CSU if you have a female loopback plug to receive the T-1 cable that was plugged into your CSU.

If you can loop the T-1 jack

Successfully looping your T-1 jack or the end of the cabling that leads to your CSU indicates that you have good physical continuity on your circuit to that point. If you are troubleshooting an issue where your carrier can’t reach your CSU, you probably have a defective CSU. If you had quality issues and the loopback tests came back clean to the hard loop on the RJ-45 jack, you again know that any static or frame slips you are experiencing aren’t being generated by the wiring. Your next phase of testing is asking your hardware vendor to validate or replace your CSU, because that is the last possible variable causing your issue.

If you can’t loop the T-1 jack

You are facing an interesting situation; you can loop the NIU but you can’t loop the T-1 jack. There are only two variables between these two pieces of hardware:

• The first variable is the cabling that connects them.

• The second variable is the T-1 jack on your NIU into which that cabling is connected.

Follow these steps:

1. Find your NIU (remember, it might be in a locked phone room somewhere else in your building).

2. Place your male loopback plug in the jack where the cabling for your circuit begins.

   Depending on the results of the test, you have two scenarios:

   If your carrier can see the loopback plug and the NIU is good, the cabling is defective between it and your T-1 jack. In this case, you need to have the person who pulled that cabling dispatched to repair it.

   If your carrier can’t see that loopback plug, the jack you are plugged into is defective and the local carrier needs to replace the NIU.

Step 5: Looping the CFA point

The CFA point is the physical point where your long-distance carrier connects into the local carrier’s network. At this point in troubleshooting, you can’t easily segment the circuit, so you need to determine whether the issue is with your carrier or with your local loop provider.

Figure 13-6 shows how the CFA point acts as the demarcation point (or demarc) between your carrier and the local loop provider.

Figure 13-6: Dedicated circuit testing: looping the CFA.

If your carrier can loop its CFA point

If your carrier can loop the CFA point and run a clean test to it, but the carrier notes errors when running to the NIU, the problem is with the local carrier section of the circuit. If your long-distance carrier ordered the local loop, then one of its representatives has to open a trouble ticket with the local loop provider. (If you ordered the local loop, you have to open the trouble ticket on your own). This process can take four or more hours before the local loop provider responds to the problem and dispatches a technician. If the trouble ticket is opened late in the day, a technician might not be available until the following business day.

If you can’t loop the CFA

If your carrier can’t loop the CFA point, the issue lies within the carrier’s section of the circuit. Because the problem is contained within one carrier, you could see a much faster resolution. Simply call the technician every hour or so to follow up until the situation is resolved.