START AT THE BEGINNING

25. START AT THE BEGINNING We're not talking about saving a few minutes by doing basic troubleshooting instead of guessing. We're talking about many hours of wasted time, when you could have found the problem quickly by just starting at the beginning (I speak from experience!). In order to start at the beginning, you need to have a clear picture in your mind of how the gizmo you're fixing is connected, so you know where the beginning is and the next thing to check. To fix a phone line, it might be the telephone company or channel bank to the NI (Network Interface), to the lightning protector and ground, to a mod jack, to the CO line card, to the CO line (or system wide) programming on the system for things like attenuation, impedance, echo cancellation, and CPC, etc.

26. You also need to think about any external stuff connected to the phone system, like the AC power, battery backup, ground, other phone lines, stations, paging etc. All of the external gizmos connected to a phone system can be causing a problem on the phone system - so you want to get them disconnected to make absolutely sure they're not causing your problem. If you leave out something, Murphy's Law says that will be the problem (that you'll find many hours later). Making yourself a list or drawing a picture of how everything is connected might be a good idea when you walk into the job

27. Simply stripping a system down and adding everything back one thing at a time until the problem returns, is the best way to troubleshoot a problem. I know you don't want to do this. You'd like to poke around at the easiest stuff first. You can do that, but if you find the problem it will usually take longer than if you started at the beginning and did your troubleshooting in a methodical manner. Fixing a system today by stripping it down and starting at the beginning is much easier than in the old days of 1A2 (with 25 to 100 pair cable). On a 1A2 system, you could have 30 or more phones that were all the lines were bridged together. If a line wasn't working right, you could disconnect the line card in the 1A2 system from all of the phones by pulling one set of jumpers. If the problem went away, one (or more?) of those 30 phones had a problem (often a water damaged 25 pair connector that got wet when mopped).

28. You could guess at which phone it was and start taking apart connectors to see if they were green inside (you couldn't see the water damage from the outside), and if you got it on the first five tries you would have gone out and played the lottery. It was your lucky day. The only reasonable way to find the problem was to start removing the 3 pair jumpers to a line on a phone, one at a time (marking them so you could put them back in the same position on the 66B block). Remove a jumper, test to see if the problem cleared, and if it didn't go on to the next jumper. If it cleared, you trace that jumper to the cable, then look at the floor plan to see where the cable was. If you were having a really unlucky day, the bad phone was the 30th jumper.

29. Modern systems are much easier! So you've got a strange problem. Since you've probably got dozens or hundreds of the particular model of system you're working on installed, and the problem isn't occurring at those other places, there must be something different about the place you're at. All you have to do is find what's different! Phone system hardware (from traditional phone system manufacturers) is more dependable than ever. The cards and power supplies just don't break like they did in the 80's and 90's. While the problem you're looking at might be caused by a bad card or power supply, it may just be an interaction with something that's connected to that particular phone system, at that particular site.

30. Since the phone system by nature has to connect to the outside world, the interface to the outside world is likely to be causing some problems. It's easy to check the power on a system by simply running the system on the battery backup to see if the problem clears (it's not a bad idea to carry a cheap battery backup in your truck for testing). Removing the system ground is also a good thing to do. The AC ground is already disconnected as soon as you pull the plug from the wall on the battery backup/UPS, to run the system on batteries. Unless you put it in yourself, it's hard to know whether a ground is good! Just because it's a green wire, it doesn't mean it's actually connected to a good ground, or even anything at the other end. Fixing an odd problem could be as simple as reversing the tip and ring on a CO line (especially from a channel bank or VoIP box), an external page port, external bells, or an MOH/BGM port. Ground loops can cause strange problems. Leaving one external device connected to a system that you thought you had stripped down could cost you hours. Although it might seem logical to assume something in your troubleshooting process (like "I've never seen one of those go bad" or even "The AC outlet must be wired right"), assuming anything without checking it in its turn in the troubleshooting process has led a lot of guys to chase their tails for hours or days.

31. We can all remember chasing our tails!  Just about every time I've assumed something, I got sidetracked and the problem took longer to fix. Sure, I still do it sometimes (it's human nature), but not very often. While it might sound harder to always Start at the Beginning, you get used to troubleshooting that way and wonder why you wasted all that time in the past! One other thing...Always try your new tester at your office or home before trying to use it at the customer's site! Once you know what your meter readings are supposed to look like, you'll feel confident using testers to solve your customer's problems.

32. If you remember, believe, and practice what you've just read...you'll be able to fix anything! I've walked into a lot of jobs where the system was down, and the customer was going nuts. If I took the queue and started acting nervous and stressed, things would have gotten worse. When I go on a service call the only thing I know for sure is that the problem can be fixed. There's no question about it. That pretty much gets rid of the stress for me, even if the customer is acting nuts. On the drive over to the service call, I get a picture of the system in my mind, and think about where to start and how to proceed. I do get screwed up occasionally when the person who took the service call wrote down the wrong problem, or didn't ask enough questions. Sometimes I'll call the customer when I'm on the way just to verify the problem (and let them know I'll be there shortly). Sometimes I'll go pick up some parts before I go on the call. The amount of traffic in bigger cities makes it pretty tough to go back to the shop for parts, as it would in a more rural area where it might be a 100 or more mile drive.

33. I personally would never agree to service a system that I didn't have spares for. If you need the spares from the trade-in on a system you're proposing, that's a good reason to give someone a good price on the trade-in. It's still going to be pennies on the dollar. Considering how many used systems are out there from brokers (or ebay), having a whole working system (with at least one station and trunk card) as a spare isn't expensive. It turns a stressful job (telephone repair) into a fun job. You don't need to have every software revision, just something to do troubleshooting and substitution with, and something to get the customer up and running until you get the right software version. Most Interconnects can't have full systems as truck stock for every type of system in your base. If you have a few trucks, it's not a bad idea to keep one type on each truck, so they can bring it over where needed. Making a deal with a cab company or delivery service to deliver a KSU, common cards, and power supply to a job site in an emergency is also a good idea (as it is when you forget something on a cutover).

34. Every Interconnect has to deal with the problem of bad cards getting mixed up with good ones. In the heat of battle, while you're swapping cards, it's not hard to lose track of them since they all look the same. I carry both gray electrical tape, and tags with string (don't use tags with wire!). The gray tape is easy to stick on a card or the bottom of a phone, so I can write down what I did and the date and time I did it when I swap something. If there isn't already a tag on a card I take out of truck stock, I put some tape or a tag on it so I know it was from the truck. If you do lose track of which card is which, having the whole system lets you test the cards in your shop before returning them to stock. There's nothing that screws up a repairman more than having bad cards for spares. It's something to avoid at all costs. I get a lot of calls from guys asking me all of the possible things that could cause a problem before they even go on a service call. Then they want me to give them the solutions to each possible problem. That would be really hard! You're causing yourself an incredible headache to do that before a service call. You've got to start at the beginning, find the cause of the symptom, and then start trying to figure out what's causing it. Every service call I go on is a challenge, but it's fun because I know it'll be fixed soon, and I get to go on to the next job. It's kind of like being a detective and surgeon in one job - but a lot safer!  What scares me is seeing a doctor that doesn't know how to troubleshoot... And there are a lot of them out there.

35. Troubleshooting Telephone Lines Disconnect the phone line from the equipment and test the line with your Butt-set. That's basic troubleshooting, but there's more reasons to do it than just to see if it's the Phone Company's problem. A Butt-set (or old 2500 set) is line powered, so it has no reference to ground or AC. That's important, since some of the problems you encounter will be an interaction between the phone line and the telephone equipment, caused by the telephone equipment's reference to ground and "foreign" voltages (like 110VAC). Every phone system will be slightly imbalanced. That means either the tip or the ring is longer than the other side of the pair, or it can have a small foreign voltage on it from being connected to the phone system. In most cases, this doesn't cause a problem. In some, it does - and those are the cases you'll be on. Calling the Phone Company and asking them to "Check the Line" or take readings for you is generally a waste of time. They're trained to say "It tests OK" no matter what. If there's a problem with a Phone Company (CLEC or ILEC) line, it's generally your responsibility to tell the Phone Company exactly why the line is bad. If you give them the numbers, they generally won't bullshit you. Hey, if everybody could do this, they wouldn't need us!

36. A common trick from Phone Company repairmen is to tell you that your phone system is imbalanced or putting out a foreign voltage or ground.. The testers they use are made for testing lines that aren't terminated. When they hook up their meter (that's made for testing unterminated lines), they're reading the phone system as well as the copper wire. They're supposed to open the line going into the premise before doing most tests so they don't read the CPE (phone system) - which will always read something strange compared to a regular 2500 set (that has no reference to ground or foreign voltage). I've seen Phone Company guys tell the customer they're disconnecting the line going into the building because it's damaging the telephone network. What a bunch of bull. Some of their test equipment will "stress test" the line, which puts out a pretty high voltage/current to try to cure problems with a copper cable. Although that usually won't hurt a phone or system, they're supposed to disconnect the CPE before they do the test. As you can guess, most Phone Company repairmen don't disconnect the CPE. They don't care. They don't have to. Fixing phone line problems is usually as simple as finding what's different about that line(s) that's having the problem, compared to others working in that system at that or at another premise. It's as simple as taking some readings (testing), and comparing the results to other lines at that premise, or at your other customers' sites.

37. Don't replace the KSU first! This is what I would do first on a strange case of trouble. I've seen a lot of strange stuff, but there's no way I'm smart enough to fix a strange problem without taking some measurements to lead me in the right direction. Once I find the solution, I might try that on the next case of trouble that's exactly the same - but you can really chase your tail for hours if you don't take a couple of minutes to take some basic readings.

39. It's as easy as taking the readings for on and off-hook DC voltage, loop current, AC, CPC, and Circuit Loss, and writing the results down on a piece of paper. This lets you compare the results to other lines at the same premise, and at other (working) sites. The problem will probably just jump out at you, and you'll know what to look at to fix the problem. Filling out this chart is very important these days because customers are switching from the LEC (or ILEC - Incumbent Local Exchange Carrier, or the real Phone Company) to a CLEC (Competitive Local Exchange Carrier, like Mike's Phone Company). In most cases, the customer doesn't call you up to ask if it will work OK. They make the change to save a few bucks, and call you when things don't work right (and then spend the next two years' savings on your T&M). By having a chart of the readings for the old LEC, you'll have a basis of comparison when the customer switches the lines to a CLEC and they don't work right.

40. Instructions: The only way to diagnose many problems on phone lines is to take voltage, current and circuit loss readings. When you have a problem, by taking readings on all of the phone lines at a premise, and comparing the results to those at other premises with no problems, the cause of the problem should jump out at you. Always take readings at premises where you aren’t having problems first, so you have a basis of comparison, and a comfort level with your readings. Never use a tester for the first time at a premise with a problem! In most cases, if you ask the phone company to take these readings, they will simply say “they’re fine.” They don’t care. They don’t have to. It’s your customer, your problem, and your money. Once you’re armed with readings that may point to the cause of the symptoms at a particular premise, going to battle with the phone company is a lot easier. Even with these reading, you may have to escalate the case of trouble, since the repair people at the phone company may not know what these readings mean.

41. NOTES: • When taking Circuit Loss readings for a “Can’t Hear” or “Can’t be Heard” problem, you must use the 1KC (milliwatt) tone for the local CO these lines are working from. If the lines are from a CLEC or on portability, you must find the 1KC tone number located at the CO these lines originate from. Using any other 1KC number will give you incorrect circuit loss readings and be meaningless, since there is always an unknown amount of loss between Cos If there are three or more phone lines at a premise and you have a 1KC Tone Generator, you can do a “Loop Around” test, sending tone on one line, and receiving it on the others. Doing some math, you can get within about 10% of the actual circuit loss on the lines. • If the there is more than 8.5db of loss on a line, the phone company has to bring it up over -8.5db (except at true Rural phone companies, where they can do anything they want to get you dial tone). Since “Can’t Hear” problems start at around -7.5db, you’ll have to change the line to a trunk (with a minimum -5.5db spec), or find a way to amplify the line yourself. • 2. The phone company doesn’t have to bring the Loop Current on a phone line down unless it’s over 110maDC. Since problems start occurring above 35ma (sometimes over 27ma), you’ll have to reduce the Loop Current yourself. If the current is below 23ma, it’s the phone company’s responsibility to get the current up over 23ma (except at true Rural phone companies).

42. NOTES: 3. The phone company doesn’t have to bring the AC on a phone line down unless it’s over 50VAC (for safety reasons). Since problems start occurring on telephone systems above .5VAC, you’ll have to reduce the AC yourself. 4. Some phone lines give a brief open loop (0VDC) when the line first goes off-hook, or right after the last digit is dialed. This can cause a cut-off if the CPC (Open Loop Timer) in the phone system is set too low. The CPC signal is normally 550 to 850ms at 0 volts, so the CPC or Open Loop Disconnect timer on a phone system should be set to 500ms (or shorter than the measured CPC signal). 5. Echo problems are usually caused by an impedance mismatch between the trunk and station equipment. Since there is no easy way to measure the impedance of the phone line or phone in the field, the only way to check is to change the impedance to see if the problem clears (using a 600 to 900 ohm Impedance Matcher).

43. You should also check the programming or jumper options for the trunks or trunk cards. You'd want to check the Open Loop Disconnect (CPC) time, any impedance settings (like 600 or 900 ohms), attenuation settings, echo canceller settings, and anything else that can effect the operation of a trunk. You will need to know the location of a Central Office, the type of CO, who owns the CO, or even how far the premise is from the local CO In my opinion, you should automatically fill out the Telephone Line Diagnostic Table for every new installation, whether you're having problems or not. I actually recommend taking the time to do it before you propose a system, so you know if you're in for trouble before you quote the new system!   It won't take long before it will take you just a few minutes to fill out the chart, and potential (expensive) problems will just jump out at you. My suggestion is to use an Installation Checklist before every job you propose. Customer's can have expectations about their phone system that they forget to tell you, or your system may just not do something the customer assumes it can do. Using the Installation Checklist can really prevent arguments, especially the ones where the customer says he won't pay you!

44. If you're not excited and stressed (or don't show it), even with the customer breathing down your neck, the customer will have less stress and probably let you do your work. If you explain to the customer what you're doing (show him your chart?), and why you're going about it in the manner you are, that might go a ways to calm an excited customer down. And then again, some customers are just going to figure that every phone man knows magic, and should be able to fix

45. Loop Current & Circuit Loss Loop Current is the amount of electrical energy flowing through the telephone and line, as opposed to the voltage which is the force behind the energy. There is a definite correlation between the Loop Current and Line Voltage (Ohm's Law), but the loop current reading is often what indicates the problems in telephony... not the voltage reading. The carbon transmitter used in telephones has been the controlling factor for years, since it needs over 20ma to sound good.

46. Loop Current & Circuit Loss • THE PROBLEM: • . • Until about the mid 1980's, the big problem with loop current was that it was often too low. That was when the only way to get from point A to point B was a pair of copper wires. Now, with the proliferation of electronic Central Offices and electronic pair gain equipment: T1, Fiber Optics, Remote Central Offices and SLICs in every suburban and even rural area (you can see these Huts and Buried Vaults scattered around everywhere), over 90% of the problems are high loop current. • This is because the manufacturers of the "far end" pair gain equipment have adhered to a very old specification for loop current, but one that is still valid, that says between 23ma and 120ma are OK - but the CPE is much closer to the source of the talk battery than the old days. When the phone company tells you that they are within specs (while smoke wafts off your trunk cards at 80ma of loop current), they're right!

47. Loop Current & Circuit Loss Both the phone company and CO equipment manufacturers have no incentive to bring the loop current down. All they have to do is make a standard 2500 set work (which has no active electronic circuitry to burn up), and the farther out it works... the better. Until the FCC sets a new standard for high loop current (unlikely), or CPE manufacturers take account of the high loop current problem (seems unlikely), the Interconnect company will be left holding the repair bag for these problems. Most CPE equipment was designed based on low loop current problems... it works well right down to 23ma. CPE manufacturers have been really slow to try to head off the high loop current problem. In the rare case of low loop current (below 23ma), the phone company is required to bring it up to 23ma.

48. Loop Current & Circuit Loss The main problem created by high loop current is heat. The components on the trunk card or telephone that connect to a CO line with high loop current get hotter than the manufacturers planned for. When the components get hot, their specs change, which makes the circuit work differently - usually with unpredictable results. If the loop current is high enough, a component can get fried and the device will stop working instead of just having problems. Eventually, the heat from the high loop current can damage one or more components on a trunk card, shortening the life of the card. BACKGROUND: Phone wire used to be much bigger than the 24 gauge standard today, so they could carry current the long distances that needed to be traveled from the CO (Central Office). There is a definite loss of electrical energy per foot of copper wire. The closer the CO got to the subscriber's location (because there are more and more COs these days), the less there was need for the current carrying capability of the bigger wires. Since the 1960's, phone wires have gotten smaller, except in rural areas where the distance to the subscriber still requires the bigger wires with larger current carrying (and lower circuit loss) capabilities. Keep in mind that some very rural CO's operate on their own totally different voltages and currents to get the phone service to their subscribers (like 100VDC talk battery).

49. Loop Current & Circuit Loss At the end of a long loop, the current that left the CO at 35ma might be 18ma because of wire loss (resistance), and the audio level may be well below -8.5db, causing it to be hard to hear. The circuit loss (volume) can be given a boost by the phone company by putting load coils into the loop, usually on loops over 3 miles. The load coil uses inductance to increase the audio voltage (sound level) at voice frequencies, while rolling off higher frequencies. If you need to use the higher frequency capabilities of the phone line for high speed data (like a 56K modem or DSL), you may have to get the load coil removed to allow those higher frequencies to pass through (although the voice capabilities of the line may then be unusable due to the distance). Data circuits have always been adjusted 13 db below voice levels (they like lines with low volume). In some cases, the Phone Company screws up and there's a misplaced load coil on the pair, either too close to the premise, too close to the CO, or not needed at all. Although everything is computerized, a lot of the plant records indicating which pairs have had load coils put on them in the past are incorrect. That means that when someone gives up a line that needed a load coil, and the pair is re-used, the Phone Company might not know the load coil is on that pair, and it could create a problem on the new phone line that pair is used for. Mis-placed load coils will change the impedance of the line and could even make it too loud, which can distort both voice and DTMF digits (causing voicemail DTMF recognition problems).

50. Loop Current & Circuit Loss A special TDR (Time Domain Reflectometer) can see the load coils on a phone line, and pinpoint the exact number of feet from the premise. If you're having strange problems, especially with DTMF recognition, it's not a bad idea to ask the phone company to check the line for mis-placed load coils. Most TDRs can see out to the first load coil on a line. A special TDR (more expensive) can see out to 5 or more load coils. Phones Companies use another piece of equipment that causes strange problems... Pair Gain Equipment. Phone companies today have little or no interest in putting more copper in the ground. If a premise has a six pair going to it from the nearest pedestal, and the subscriber needs seven lines, the Phone Company has two choices, bury more copper cable, or use an electronic gizmo to get two lines from one of the existing pairs.