Getting the Most from Your Carriers TroubleshootingDepartment

Getting the Most from Your Carrier’s Troubleshooting Department

Most carriers have a two-tier structure for handling problems. The first tier of people you meet are the entry-level customer service folks. These people generally work from a script that asks you specific questions to qualify your issue. After a customer service agent has all the information he or she needs, you’re given a trouble ticket number for tracking purposes.

If the first tier of customer service can’t resolve the problem, the trouble ticket is sent to the network technicians, who make up the next level of support. These people can manipulate the network, update switches, perform tests, and are empowered to fix the complex things that go wrong. These are the people you want to speak with when you have a complex issue.

 Tip  If you have a difficult and intricate issue, give the first-level customer service people just enough information to open the ticket and ask to chat with a technician. If you begin trying to tell the customer service rep all the minutiae of the problem, it might get lost in translation from you, to them, to the notes, to the tech. It’s always safer to simply get the ticket open, press for a tech, and then explain it all to the tech.

There are three points to keep in mind when chatting with the first-tier customer service staff:

• The rep you talk to is probably working off a script that has required information. You can’t proceed without going through this stuff. If you don’t have all the information you need, things won’t go very far, so make sure you gather as many details as you can before you pick up the phone. For example, if you have a problem dialing a phone number in Alaska, but you don’t know the number you dialed because you lost your notes, the carrier can’t open the trouble ticket.

• All the information you give guides how your problem is handled. If you tell the customer care rep that your dedicated circuit can’t dial to area code 414, the ticket will be sent to the dedicated department to test your T-1 lines. If the same problem happens when dialing phone numbers in the 414 area code from your switched phone lines, the trouble ticket is sent to the department in your carrier devoted to handling calls from the PSTN in the 414 area code to determine why that area code is failing.

   Remember  It is always preferable to open a trouble ticket as an outbound switched problem if it affects that call type. This option is always the most direct route to resolution and prevents your carrier from being distracted by inconsequential aspects of the problem. The same holds true for an issue that affects both your inbound and outbound toll-free calls to a specific area. Opening a trouble ticket as a toll-free problem focuses your carrier on the toll-free aspect of the call (SMS database construction, areas of coverage, and nuances that are particular to toll-free numbers), and not on the overall network (which is actually the source of the problem).

• The person who opens your ticket is one of your greatest allies. He or she can escalate the ticket on your behalf, monitor its progress, and call in favors to resolve your issue. Because the customer service rep is capable of doing so much for you (and has the power to do nothing at all), always be nice. If the customer service person you are chatting with doesn’t understand what you are saying, graciously ask for a supervisor.

Understanding the translating and routing of calls

 Technical Stuff  Every carrier receiving a call must determine where to send the call; then it must send it. These two activities represent two of the most common areas causing the failure of calls. Translation is the process a carrier undertakes to identify the destination of the network that must receive the call. Long-distance networks don’t send calls all the way to the end telephone receiving the call; the local carrier handles that job. The long-distance carrier sends the call to the local carrier’s specified central office (CO). If the local carrier decides to use a different CO to ring your phone, but your long-distance carrier is not aware of the change, your long-distance carrier sends the call to the wrong CO and the call fails. This problem is corrected by the translations department at your long-distance carrier, because the translations of the number you dialed to the correct CO must be resolved.

When a carrier’s network determines where to terminate a call (by choosing the correct local carrier CO) the carrier has to deliver the call to that location. The route is the path the call takes from the moment it enters the carrier’s network to the point it leaves the carrier’s network. The routing department within each carrier monitors these routes and prevents them from being congested. If one main circuit fails, calls proceed through a secondary or tertiary route set up by the routing department. If a large outage occurs, or calls fail on their way to the correct local carrier CO, the routing department identifies the fault in the network and repairs it.

Identifying your call treatment

When you call to report a problem with your phone service, the rep at your carrier asks you to describe it. The more specifically you can describe the issue, the easier it is to find and repair. The rep might directly ask you for the call treatment, or the symptoms of the failed, or substandard call. This is where telecom can seem like a foreign language. You know what is happening, but putting it into words that can be understood might not be that simple. The main call treatments you will encounter are listed in the following sections.

Understanding why “your call cannot be completed as dialed”

If you hear a recording that tells you that your call can’t be completed as dialed, you may have misdialed the number, the area code of the number you are dialing having changed, or a translation problem may have occurred in a carrier network somewhere along the line. This recording generally indicates that the problem affects an individual phone number. Your ability to dial other numbers in the state, town, or country is usually not impacted. Try the number again, check the digits and the area code, and then make the call to your carrier’s customer service.

Understanding why “the number you have called has been disconnected or is no longer in service”

A recording that says that the number has been disconnected or is no longer in service might be legitimate; the number has simply been disconnected. If you know it’s not disconnected, either you misdialed the number, or there is a translation issue somewhere.

Handling an “all circuits are busy” message

On rare occasions you may hear a recording that tells you that all circuits are busy. This recording rarely means that all the circuits available in your carrier’s network are occupied, unless you are trying to call your Mom at 9 a.m. on Mother’s Day, along with everyone else in the world. This recording is generally played when your carrier has an outage of some sort and a portion of their network is down. If a backhoe accidentally cuts through a phone line, thereby taking down phone service for the entire city, you will probably hear this recording when you try to dial out (the carrier unfortunately doesn’t have a sorry, but our main phone line has been cut by a backhoe recording).

Listening for tones and tags

Tones and tags are supplemental sounds or recordings and are generally attached to a standard recording. Listen for tri-tones that are played before recordings by a local carrier. They are generally three ascending notes that sound like they come from a cheap synthesizer — very shrill and high pitched.

The tags are more important because they frequently list the switch that is playing the message. If you hear a recording of your call cannot be completed as dialed — fifteen dash two, you know that switch 15 on your carrier’s network probably has a problem. If you hear tones and tags during a recording, be sure to note them, because alerting a technician to the specific tags can shorten the amount of time it takes to solve your problem. Instead of tracking down the failed call, the technicians can go directly to switch 15 and analyze it. The number after the dash may correspond to the recording played (for example, the two in this example may mean cannot be completed as dialed). On the other hand, the number may have no significance at all. Every carrier has its own system for the tags played. The tags may mean nothing to some carriers, but to others they can be significant. I say you should always have more information than less.

Understanding the fast busy signal

A fast busy signal is a busy signal that sounds twice as fast as the normal busy signal. You will probably hear a fast busy signal when part of your carrier’s network is down (the pesky backhoe again — see “Handling an ‘all circuits are busy’ message,” earlier in this chapter), so your call can’t be completed.

Handling echo, echo, echo

Usually, only one person on a call hears an echo. Carriers have specific pieces of hardware installed throughout their network called echo cancellers (or echo cans) to eliminate echoes on calls. These devices can fail over time, be misoptioned, or mistakenly installed backwards. The interesting thing is that if one person hears an echo on a call, it’s probably the result of a bad echo can on the other side of the call. Sometimes, both people can hear the echo, but it’s not as common.

Echo doesn’t manifest itself in a way that’s immediately visible to the technicians at your carrier. If your call fails to a fast busy signal, your carrier can pull the call record and find the switch that killed the call. Some issues, such as dropped calls or static, are visible in a circuit’s performance report, which indicates that there has been an electrical or protocol-related anomaly. Echo on a call, however, doesn’t leave a trail of breadcrumbs and is therefore a difficult issue to isolate and repair.

The reverberation of voice that people hear and classify as echo might also be the result of something other than a failing or mis-optioned echo can. This other type of echo is referred to as hot levels, and is the result of the over-amplification of a voice in the transmission of the call. If you didn’t have an equalizer in your stereo and you cranked the volume up as far as it could go, your sound quality would be horrible and you would experience reverberation (or what you might call echo). To compensate for volume loss as calls are transmitted across hundreds of miles over circuits within the network, the switches in the POPs will boost the sound a fraction of a decibel, what is called padding the call, to keep the signal strong. The problem is that if the carrier amplifies the sound too much, you will hear echo. What is worse is that hot levels have cumulative effects.

 Remember  If a call travels 15 miles and is padded 0.4 decibels in the two pieces of hardware it encounters, you won’t hear echo. In fact, as long as the call has fewer than two decibels of padding from start to finish, you won’t have a problem. If the call is traveling from coast to coast and each of the six long-distance POPs bumps the volume up 0.4 decibels, the call now has 3.2 additional decibels in padding. You can bet you will hear echo. The cumulative nature of hot levels makes it difficult to find and correct. In situations like this, the issue might persist for weeks before the carrier finds all the offending switches and reduces the padding.

Standing up to static

Only one side of a call usually hears static, but it does affect both incoming and outgoing calls. It’s generally caused by a piece of hardware slowly dying in the network. The static can be very minor to begin with, but grows over time until you can’t hear the person you’re calling, or they can’t hear you. As the issue evolves, the piece of hardware will eventually die and all of your calls will fail. Don’t feel bad; the bigger the issue, the easier it is to find. When the static is noticeable on every call, it becomes simple to track it down and replace the affected hardware.

 Technical Stuff  Echo might be invisible to the technicians monitoring a network, but generally static isn’t. Almost all circuits in the U.S. are capable of being monitored for quality. This doesn’t mean someone from your phone company is listening in on calls to ensure that they sound clean, but computer files can capture any unexpected electronic or protocol activity on a circuit. Static can be classified as unexpected electric activity, as it’s some device experiencing an electrical short. The performance monitors (PMs) are diagnostic files for a circuit that record the electric and protocol anomalies. The challenge with static is finding the correct span causing the problem. If you’re dealing with an intermittent issue affecting five percent of your calls somewhere in the switched network for your carrier, the possible locations of the failing hardware can seem endless.

Dealing with dead air

Dead air is a term that refers to the phenomenon when you hear nothing on the other end after you dial a phone number. You don’t hear the dial tone anymore, but you also don’t hear any ringing; you just hear nothing. When you hear dead air, stay on the call for 30 to 60 seconds; you’ll probably hear a fast busy signal if you wait long enough. Dead air is generally caused by a translation or routing problem that caused your call to be transferred to a piece of hardware or a circuit that no longer exists. Because the hardware no longer exists, nothing is there to send you a polite recording, busy signal, or anything. All you get is dead air.

 Remember  Dead air isn’t the same thing as post dial delay (PDD). PDD is the silence you hear for a few seconds before you hear the ringing. Every call has some PDD, albeit it might last only one second. International calls are notorious for long PDD; anywhere from 15to 30 seconds may pass before you hear the phone ring on the other end.

Getting around clipping

Clipping is the technical opposite of echo. With echo, a voice is repeated; with clipping, sections of the voice transmission are lost. Instead of hearing the conversation in full, you hear only sections of each word, because a few milliseconds are lost every second. This problem is solved just like echo or hot levels, but instead of having too much volume on the call, you have insufficient volume. The troubleshooting process for clipping is the same as any quality issue and requires both patience and a large quantity of call examples.

Dancing around dropped calls

Phone calls that are disconnected before either person hangs up are deemed dropped calls. If your phone system loses power while you are talking, it will drop your call. The same thing happens if you are calling over a dedicated circuit that suddenly fails or takes an electrical hit. Dropped calls are researched by your carrier and the exact cause of the disconnection is identified by a disconnect code passed through the network when the call ends. The disconnect code identifies whether the call was dropped by the origination side of the call, the termination side of the call, or an unknown event in the carrier’s network. Dropped calls are generally the result of a failing piece of hardware and typically become more frequent until the source of the problem is evident. If your carrier is causing the disconnections, its technicians will find the failing hardware by protocol failures in the protocol monitors of the circuit.

Handling aberrant recordings

Each carrier has a few standard recordings; you don’t hear a ton of variety. Any other recordings you hear probably come from your phone system. Your carrier doesn’t have any recordings on file that say, “We regret that you were unable to access an outside line,” or “Your long-distance carrier is currently rejecting your call.”

If you receive any recording that refers to your carrier, it probably was not made by the carrier’s network. You might want to check with the carrier first to ensure the message isn’t in its play list, but then you need to investigate, either with your phone system, or with the phone system of the number you dialed.

Working around incomplete dialing sequences

If you dial a long-distance phone number and the call is dropped after you press six or seven of the digits, you have two possible sources to check:

• Incomplete dialing sequence caused by local carrier: If your local carrier perceives that the number you are dialing is a local call, it might try to complete it after the seventh digit. If the call is actually a long-distance one, you need to take this issue up with your local carrier immediately.

• Incomplete dialing sequence caused by your phone system: This situation can also occur on some phone systems when international numbers are dialed. Your phone system might limit you to dialing ten digits. Because the international prefix of 011 already takes up three digits, and then possibly two to six digits for just the country and city code, you can easily reach the limit set by your phone system. Your call is dropped even before it leaves the office. You need to talk to your hardware vendor so that your phone configuration can be changed.

   Remember  If you are dialing from a switched phone line, you must be able to complete dialing all the digits of a phone number before your local carrier processes your call and sends it to your long-distance carrier. If you have a dedicated circuit, check your phone system and then contact your long-distance carrier. The long-distance carrier can watch you dial the phone number and see every digit you enter. More important, the technician can determine whether the hang up on the call is being initiated by your phone system or by the network. When you know the cause of the disconnection, you know where to turn to resolve it.

Providing a call example

A call example contains detailed information that allows your carrier to follow the call’s path from the moment you first dialed the number to the point the call failed. As technical as the idea sounds, a call example is just the information you write down about a failed call. After you dial out and get a cannot be completed as dialed recording, dial the number again and write down the necessary information (see the following sections for more information about what to include in your notes). When the carrier finds the call’s end point, the technician can begin correcting the issue.

Call examples function to not only tell the technicians where to look for the problem, but also to allow the customer service rep to categorize the issue. Depending on the information you provide, the customer service rep will send your issue to a specific department for repair.

Call examples might not be easy to come by in some instances. If you are calling a number you dial often and the call fails, you will know all the information required to open a trouble ticket. The challenge comes in when customers dialing in on your toll-free number have an issue. Customers might not have your direct phone line to tell you that they couldn’t get through and report the issue. Even if they do get through to you, it’s not common to begin your conversation with a quiz about the specifics of a failed call attempt. As a result, you might have to ask one of your customers to make test calls for you. The specific information your carrier needs is listed in the following sections.

 Remember  Call examples have a shelf life of about 24 hours. The specific information about how a call is routed is kept in your carrier’s switches for a finite amount of time before it’s overwritten with new, more recent calls. If an issue crops up on Friday at 5 p.m., you need your carrier immediately. If you try to provide the call example from Friday when you come into the office on Monday, your carrier will probably reject it and ask you to give them a newer one.

The date and time of call

Every call is logged into your carrier’s switches by the origination time. If you made four calls to a phone number and only two of them failed, be sure to give your technician as much information as possible to help differentiate the completed calls from the failed ones. If you provide one call example and don’t mention the other three attempts, the technician might find one of the completed calls and close the ticket because their research indicated that the call didn’t fail.

 Remember  There are four time zones in North America, plus Alaska and Hawaii, so be sure to identify the time zone when you provide a call example. If you don’t tell your carrier that the call was made at 8 a.m. eastern standard time, the customer service person might record the time by using a local time zone.

The origination phone number

You might be one of 10,000 people who call a specific phone number every day. The only way to isolate your call from all the others is by referencing it back to your phone number.

 Tip  If you are calling from a phone system that randomly assigns a line as you dial out, you might not know the originating phone number. This isn’t a problem. Generally, the outbound phone lines you dial from are provided in sequence, so simply use any of the numbers as your origination, and then tell your technician that you are dialing from a phone system. The technician will note this fact in the ticket. When the call examples are found, the technician can trace the call back to your office by matching the area code and the first three digits of your phone number.

The number you dialed and the call treatment

If you made 5,000 calls in the past three hours, and one of them failed, your carrier needs to know which one out of the 5,000 numbers you dialed was the problem child. Seems reasonable, yes? The phone number you dialed gives your carrier an idea of the geographic area you are dialing into and is essential for tracking down the problem.

 Tip  See “Identifying your call treatment,” earlier in this chapter, for more on what information you should include when you describe the call treatment. Saying that the phone call “just didn’t work” . . . just doesn’t work.

Understanding when to provide multiple call examples

A phone call has many paths it can take to reach its destination. Depending on fluctuations in the capacity of the network between the two points, you could make 10 calls to the same number in 15 minutes and your calls may never take the exact same route twice. Understanding the complexities of the phone systems is the key to resolving intermittent issues. The larger a problem, the easier it is to track down and repair. If you hear dead air, static, or echo on 5 percent of all your calls, you need to provide as many call examples as possible.

Whenever you have an intermittent issue, it’s very helpful to provide to your carrier clean call examples in addition to problem calls. This will allow the technician to review all the calls and begin comparing the individual circuits the calls took. After the technician eliminates all the circuits on the clean calls and isolates any remaining similar circuits on the affected calls, he or she will be more than halfway through troubleshooting your problem.