280 likes | 552 Vues
Low Level Commissioning 2005/2006 Collaboration Meeting, Baton Rouge Mark Krasberg April 12 th , 2006. Commissioning a DOM at The Testing DAQ Level. Commissioning plan pretty much followed. Could not study string 29 during freeze-in
E N D
Low Level Commissioning 2005/2006Collaboration Meeting,Baton RougeMark KrasbergApril 12th, 2006 Mark Krasberg U of Wisconsin
Commissioning a DOMat The Testing DAQ Level Mark Krasberg U of Wisconsin
Commissioning plan pretty much followed • Could not study string 29 during freeze-in • Ended up taking more data than original plan (not a bad thing) • Cold temperature problem meant that not all elements of the plan could be followed Mark Krasberg U of Wisconsin
Problem Quads • Leaking Quads • 12/25 String 29 all quads • 01/04 String 39 none (quad 7 SJB problem) • 01/08 String 38 leak-checked late (oscillations) • 01/14 String 30 quads 4, 7 (quad 14 SJB prob) • 01/18 String 40 quads 13 and 14 • 01/22 String 50 leak-checked late (damaged) • 01/25 String 59 leak-checked late (swamped) • 01/29 String 49 quads 6, 9 and 10 • Anomolies • Any DOM which exhibited anomolous behavior was unplugged • Several wire pairs had high currents and their quads were unplugged and periodically tested • Communications Issues • Encountered a cold temperature (cold reboot) problem • Issue was a moving target as DOMs cooled and changed temperatures • Problem seems to be understood (final fix not applied at Pole yet) Mark Krasberg U of Wisconsin
String 39 two-week freeze-in movie Mark Krasberg U of Wisconsin
Triboluminescence • Had a longstanding plan on how to cope with the triboluminescence: (>40 kHz individual DOM rates!) • Use LC up.AND.down (“Triple Local Coincidence”) • Use Scaler Deadtime (reduces LC trigger rate) • The plan worked! • able to take over 1 terrabyte of data during season and hand-carry north Mark Krasberg U of Wisconsin
Data-taking during Triboluminescence Approximate maximum trigger rate for normal TestDAQ data-taking frozen-in LC rate Mark Krasberg U of Wisconsin
Surface DAQ Hardware • Jan 15: string 21 hub stopped working, probably overheated • Did not operate all hubs simultaneously after this event until the power distribution and cooling fan problems in the TCH were solved • Jan 30: string 59 DSB card didn’t work – was replaced • String 59 DOMHub suffers from domhub-app JVM crashes. This is the only hub which has had more than one JVM crash with TestDAQ. RealDAQ has had problems with other hubs. Mark Krasberg U of Wisconsin
Cold Temperature Problems • DOMs found to have temperature-related problems (“cold reboot problem”) • Problems occur primarily at top of string (colder) • THIS MADE IT IMPOSSIBLE TO COMMISSION ANY DOMs at Pole • communications problems were a moving target • the list of DOMs which would not communicate properly would change daily, as the DOMs cooled down. • Particularly problemmatic temperature is ~-28C Mark Krasberg U of Wisconsin
Study of cold reboot problem in ice • Used internal heating of mainboard to probe a temperature range (like in FATs) • String 29 “Farsta” • 57 reboots between -34.44 C and -27.25 C • 49 successful cold reboots • All reboots (8) failed between -31.25 C and -30.56 C • No successful reboots in this range • Just as during FAT testing, the problem is isolated to a specific, narrow temperature range Mark Krasberg U of Wisconsin
Effect of cold temperature problems on Commissioning Mark Krasberg U of Wisconsin
Able to deal with the expected unexpected problems fairly easily(part of the plan!) • When we hook up DOMs in a new environment we often encounter problems • Each DOMHub had a full TestDAQ development workspace (ie all the code needed for data-taking could be modified and recompiled at pole). • Patches could be emailed through Iridium satellite • Patches could then be installed, recompiled, and tested within minutes • Feedback could be given to the north almost instantaneously using Iridium satellite – we did not have to wait for TDRS satellite. If we had, then the resulting code iterations would have gone on for days! Mark Krasberg U of Wisconsin
Communications now stable • TestDAQ softboot problem eliminated March 8. • DOM communications fix waiting to be deployed at pole • Data taking became stable with TestDAQ as of March 8th (600 DOMs at that time) except for occasional string59 domhub-app crashes. • DOMs can be commissioned since the communications problems have effectively been eliminated and data-taking is stable. • AMANDA mainboards have been included in the readout in almost all TestDAQ runs since February Mark Krasberg U of Wisconsin
High Current DOMs • Six high current wire pairs identified at pole • Four of the high currents mysteriously went away (seemingly not related to temperature or freeze-in). One took weeks to go away! • Two remaining high current pairs permanently plugged in at station close • DOMs on both of these pairs have suffered problems in the months since they were plugged in • In the future, high current pairs should not be used in normal data-taking for as long as is feasible (one year?) Mark Krasberg U of Wisconsin
High Current DOM Pairs • High current pairs which are still high current pairs • | 39-07 | Hippophilia | 8b56bf63b908 | UP5P0888 | 0 | 3 | • | 39-08 | Shamal | 7b492c28a830 | TP5P0527 | 0 | 3 | • | 40-51 | Juneberry | ce9692f580c9 | UP4P0086 | 6 | 1 | • | 40-52 | Alfa_Romeo_Spider | 0a9dc8de00c3 | TP5Y0171 | 6 | 1 | • High current pairs which are no longer high current pairs: • | 49-01 | Ceres | 754dad1d3800 | UP4Y0030 | 0 | 0 | • | 49-02 | Marzahn | 60cad4124a82 | TP5Y0161 | 0 | 0 | • | 50-37 | Volkswagen_Beetle | 93c6fcb0fe68 | UP5Y0216 | 4 | 2 | • | 50-38 | Warthog | a095b31f89cf | TP4P0315 | 4 | 2 | • | 40-57 | Tortellini | d54044599535 | UP4P0140 | 7 | 0 | • | 40-58 | Cannelloni | 3dac398e0d9d | TP4P0111 | 7 | 0 | • these DOMs drew an extremely large current initially, before problem went away... • | 30-45 | Capybara | 0d42d3e12c9f | UP4P0276 | 5 | 2 | • | 30-46 | Asparagus | 44c3682eb25c | TP4P0217 | 5 | 2 | • The pair with 50-58 “Universitet” had a marginally high current at pole but was not flagged as high current – it’s LC was determined to be broken as soon as it was turned on. Mark Krasberg U of Wisconsin
Mark Krasberg U of Wisconsin
40-51 “Juneberry” & 40-52 “Alfa_Romeo_Spider” poor comms 50-58 “Universitet” bad LC 39-08 “Shamal” bad LC 59-59 “Cosmology” poor flash 29-18 “Yurei” PMT? 38-59 “Blackberry” no HV 49-17 “Biometeorology” bad flash 30-23 “Peugeot_505” no power 50-36 “Ocelot” no power 29-59 “Auroraphobia” & 29-60 “Nix” no power Mark Krasberg U of Wisconsin
There are currently 16 strings (not 9)! hole: 21 29 30 38 39 40 49 50 59 strings: 1 1 2 1 2 2 2 3 2 total=16 we had 17 mini-strings in UW FAT #6! broken LC connection between two DOMs Problem DOMs a very short string! cannot be used for hard LC Mark Krasberg U of Wisconsin
STF results for string 38, odd (U) DOMs Mark Krasberg U of Wisconsin
38-59 “Blackberry” failure Mark Krasberg U of Wisconsin
38-59 “Blackberry” failure zoom Mark Krasberg U of Wisconsin
Freeze-In: 49-55 “Fusilli”DOM was still in water on Mar 22 Mark Krasberg U of Wisconsin
String 49 – drill stopped for ~one hour at ~2300 meters Mark Krasberg U of Wisconsin
Summary of Commissioned DOMs • To date: the 300 DOMs for which STF ran successfully at Pole can be regarded as commissioned (low level) • when 49-55 “Fusilli” freezes-in STF will be run again, as per initial plan (don’t expect any problems) • Current Status: 600 out of 604 communicating on March 20 • DOMs with serious problems • Three communicating DOMs not useful for hard LC data-taking (this affects 4th DOM) • 2 DOMs (40-51 “Juneberry” and 40-52 “Alfa_Romeo_Spider”) are high current, poorly communicating, have recently degraded and should be unplugged for special testing • DOMs with less serious problems • 2 DOMs (49-17 “Biometeorology” and 59-59 “Cosmology”) have flash problems • 29-18 “Yurei” suffered a 40% gain shift in March • 21-30 “Phenol” has sustained high dark rate problem, operating at lower gain • 21-13 “Wickueler” does not like to be turned off (discovered during this season) • Additional LC problem between 59-51 and 59-52 (broken connection during refreeze) Mark Krasberg U of Wisconsin
Mark Krasberg U of Wisconsin
49-55 “Fusilli” Mark Krasberg U of Wisconsin
Mark Krasberg U of Wisconsin
Mark Krasberg U of Wisconsin