ADCP Toolbox How - To

1. ADCP ToolboxHow - To U.S. Geological Survey Woods Hole Science Center A tutorial on how to use the ADCP Data Processing SystemA tutorial on how to use the ADCP Data Processing System

2. USGS Apr. 6, 2008 What is the ADCP Toolbox? A set of MATLAB M-files that performs post processing routines on binary RDI ADCP data files Allows processing data with interactive critical review Converts the raw binary data into EPIC standard netCDF archival format

3. USGS Apr. 6, 2008 Primary Uses Fixed platform upward-looking recorded in beam coordinates Downward-looking on a fixed platform recorded in beam coordinates Toolbox can be used for data recorded in earth coordinates on a moving platform The design of the toolbox is most tailored to ADCP's deployed on fixed platform and upward-looking recorded in beam coordinates. However, the toolbox works equally well for ADCP's deployed on a fixed platform downlooking, with the exception of trimming the bottom bins. Additionally, the toolbox will convert data recorded in Earth coordinates to EPIC netCDF format, but it does not contain any routines for masking in this format.The design of the toolbox is most tailored to ADCP's deployed on fixed platform and upward-looking recorded in beam coordinates. However, the toolbox works equally well for ADCP's deployed on a fixed platform downlooking, with the exception of trimming the bottom bins. Additionally, the toolbox will convert data recorded in Earth coordinates to EPIC netCDF format, but it does not contain any routines for masking in this format.

4. USGS Apr. 6, 2008 Basic Processing Steps Step 1: Prescreen data, TRDI winadcp. Step 2: Convert, Mask and Trim, runadcp.m - convert data from binary to netCDF - automated masking of bad data based on RDI data quality standards - additional on-screen user discretion masking - trimming the bins that are out of the water Step 3: adcp2ep.m - convert file format to EPIC netCDF for ease of archival and interpretation - addition of metadata to the file There are two basic steps, or programs to the ADCP Toolbox, runadcp, and adcp2ep.There are two basic steps, or programs to the ADCP Toolbox, runadcp, and adcp2ep.

5. USGS Apr. 6, 2008 What else do I need? Current version of the netCDF Toolbox available at http://mexcdf.sourceforge.net Current version of mexnc for your OS at http://mexcdf.sourceforge.net Put the netCDF Toolbox, mexnc and the ADCP Toolbox in your MATLAB path from RDI�s website http://www.rdinstruments.com/ winadcp wavesmon if the ADCP was set up for waves Ncbrowse, from http://www.epic.noaa.gov/java/ncBrowse/ at the link http://www.rdinstruments.com/cc_software.html Surface.exe is under Broadband Tools, Data Handling Tools Winadcp is under workhorse software, Winadcp.at the link http://www.rdinstruments.com/cc_software.html Surface.exe is under Broadband Tools, Data Handling Tools Winadcp is under workhorse software, Winadcp.

6. USGS Apr. 6, 2008 Wave deployments If the ADCP was set up to measure waves Run splitadcp.m first to split out the current ensembles, see the USGS wave data processing toolbox for wave processing The wave toolbox can be downloaded at http://pubs.usgs.gov/of/2005/1211/ It give suggestions for running TRDI wavesmon and translates wavesmon output to netCDFThe wave toolbox can be downloaded at http://pubs.usgs.gov/of/2005/1211/ It give suggestions for running TRDI wavesmon and translates wavesmon output to netCDF

7. USGS Apr. 6, 2008 Screening with winadcp Determine first and last good ensembles Determine if the ADCP�s orientation setting matches the actual, deployed orientation Look for extreme pitch and roll An ADCP deployed as uplooking but thought it was downlooking will require extra attention. See Tips and Tricks documentationAn ADCP deployed as uplooking but thought it was downlooking will require extra attention. See Tips and Tricks documentation

8. USGS Apr. 6, 2008 Information needed for processing: Mooring number Deployment date Recovery date ADCP serial number Distance between transducers and seabed Predicted accuracy Clock error Magnetic variation Water Depth Deployed orientation Presence of pressure sensor Experiment name Project name Description of data Comments Longitude Latitude *.dlg file This is the information you will need prior to running the ADCP toolbox. The processing will move along much easier if you have all of this information written down in one location, as this information needs to be obtained from several sources, like the mooring log, instrument books, etc.This is the information you will need prior to running the ADCP toolbox. The processing will move along much easier if you have all of this information written down in one location, as this information needs to be obtained from several sources, like the mooring log, instrument books, etc.

9. USGS Apr. 6, 2008 Sample Data Processing Sheet We have found that a checklist sheet like the one shown here works very well for us. You should all have a copy of this in your packet, so I would like you to find it and take a look at it with me. There are two big things I like about this checklist, 1) it organizes all of the information I will need into one central location. This way, if I have to run the toolbox several times for the same data file, I don't need to go searching through the mooring logs each time to find the metadata. 2) it ensures that I perform all of the processing steps in the right order, and keeps track of what files have been created during the process. Have class fill in a blank checklist compiling information from the mooring log and data processing sheetsWe have found that a checklist sheet like the one shown here works very well for us. You should all have a copy of this in your packet, so I would like you to find it and take a look at it with me. There are two big things I like about this checklist, 1) it organizes all of the information I will need into one central location. This way, if I have to run the toolbox several times for the same data file, I don't need to go searching through the mooring logs each time to find the metadata. 2) it ensures that I perform all of the processing steps in the right order, and keeps track of what files have been created during the process. Have class fill in a blank checklist compiling information from the mooring log and data processing sheets

10. USGS Apr. 6, 2008 Sample Data Processing Checklist We have found that a checklist sheet like the one shown here works very well for us. You should all have a copy of this in your packet, so I would like you to find it and take a look at it with me. There are two big things I like about this checklist, 1) it organizes all of the information I will need into one central location. This way, if I have to run the toolbox several times for the same data file, I don't need to go searching through the mooring logs each time to find the metadata. 2) it ensures that I perform all of the processing steps in the right order, and keeps track of what files have been created during the process. Have class fill in a blank checklist compiling information from the mooring log and data processing sheetsWe have found that a checklist sheet like the one shown here works very well for us. You should all have a copy of this in your packet, so I would like you to find it and take a look at it with me. There are two big things I like about this checklist, 1) it organizes all of the information I will need into one central location. This way, if I have to run the toolbox several times for the same data file, I don't need to go searching through the mooring logs each time to find the metadata. 2) it ensures that I perform all of the processing steps in the right order, and keeps track of what files have been created during the process. Have class fill in a blank checklist compiling information from the mooring log and data processing sheets

11. USGS Apr. 6, 2008 Start MATLAB

12. USGS Apr. 6, 2008 Set the directory

13. USGS Apr. 6, 2008 Set the directory

14. USGS Apr. 6, 2008 Create a diary log file

15. USGS Apr. 6, 2008 runadcp

16. USGS Apr. 6, 2008 Choose the number of binary files to process This usually occurs when one of the memory cards fills up and then a new data file will be created on the second memory card. Also, sometimes the ADCP will hiccup and you�ll end up with two or more files. Right now, the runadcp can accommodate the combination of only two files, though there are ways to concatenate more than two if it is necessary.This usually occurs when one of the memory cards fills up and then a new data file will be created on the second memory card. Also, sometimes the ADCP will hiccup and you�ll end up with two or more files. Right now, the runadcp can accommodate the combination of only two files, though there are ways to concatenate more than two if it is necessary.

17. USGS Apr. 6, 2008 Select the Binary ADCP data file This is the file you directly upload from the ADCP memory card or through the communications cable.This is the file you directly upload from the ADCP memory card or through the communications cable.

18. USGS Apr. 6, 2008 Don�t forget to write the name of the file on the Data Processing Checklist.Don�t forget to write the name of the file on the Data Processing Checklist.

19. USGS Apr. 6, 2008 Save the netCDF file This is utilizing the rdi2cdf.m function which translates the raw binary ADCP data file into netCDF format, but no editing is done in this step. I like to keep the same name as the raw data file, and just change the extension to .cdf. Also, here we add the instrument number position to the data file. It is important to maintain a standard naming convention as it will allow you to consistently and easily trace your steps through the processing system, if need be.This is utilizing the rdi2cdf.m function which translates the raw binary ADCP data file into netCDF format, but no editing is done in this step. I like to keep the same name as the raw data file, and just change the extension to .cdf. Also, here we add the instrument number position to the data file. It is important to maintain a standard naming convention as it will allow you to consistently and easily trace your steps through the processing system, if need be.

20. USGS Apr. 6, 2008 Don�t forget to write the name of the file on the Data Processing Checklist. Don�t forget to write the name of the file on the Data Processing Checklist.

21. USGS Apr. 6, 2008 Enter mooring information

22. USGS Apr. 6, 2008 Enter the ADCP information

23. Enter the ADCP Information This is a good time to trim off data when the instrument was in the air. Use winADCP to determine the first good and last good ensemble numbers. USGS Apr. 6, 2008 This is a good time to trim off data when the instrument was in the air. Use winADCP to determine the first good and last good ensemble numbers.This is a good time to trim off data when the instrument was in the air. Use winADCP to determine the first good and last good ensemble numbers.

24. USGS Apr. 6, 2008 -Wait- Ask questions. Emphasize importance of maintaining naming conventions and following the checklist.Ask questions. Emphasize importance of maintaining naming conventions and following the checklist.

25. USGS Apr. 6, 2008 Verify the Deployed Orientation TRDI ADCPs sometimes get fooled Select the orientation in which the ADCP was actually deployed

26. USGS Apr. 6, 2008 Are these depths reasonable? I recommend always typing �N� because then you can look at the data more closely and type in your own values that might better reflect the data.I recommend always typing �N� because then you can look at the data more closely and type in your own values that might better reflect the data.

27. USGS Apr. 6, 2008 You should see something like this:

28. USGS Apr. 6, 2008 Saving the Fill file Creating a Fill file will find the ensembles in the data file that are missing, the number of the missing ensembles, and the time of the missing ensembles. It will then create a dummy array of fill values for each variable in the data set at the time of the missing ensemble numbers as place holders. I haven't had a need for this, so it may be that RDI has fixed a bug in the firmware to eliminate this occurrence.Creating a Fill file will find the ensembles in the data file that are missing, the number of the missing ensembles, and the time of the missing ensembles. It will then create a dummy array of fill values for each variable in the data set at the time of the missing ensemble numbers as place holders. I haven't had a need for this, so it may be that RDI has fixed a bug in the firmware to eliminate this occurrence.

29. USGS Apr. 6, 2008 Saving the mask file Creating a mask file *.msk will make a duplicate of the data file *.cdf which is used to identify bad data for later removal. Creating a mask file *.msk will make a duplicate of the data file *.cdf which is used to identify bad data for later removal.

30. USGS Apr. 6, 2008 Saving the Masked File Here, we are creating the name of the file that will bear all of the automated and hand-masking edits.Here, we are creating the name of the file that will bear all of the automated and hand-masking edits.

31. USGS Apr. 6, 2008 Update the data processing checklist!!!Update the data processing checklist!!!

32. USGS Apr. 6, 2008 Save the trim file as� This will create a new netCDF file that will contain the masking that was just done, and the ensemble trimming and bin trimming that are about to be performed.This will create a new netCDF file that will contain the masking that was just done, and the ensemble trimming and bin trimming that are about to be performed.

33. USGS Apr. 6, 2008 Verify when to trim in time The sub-function �goodends� attempts to figure out the beginning and end of the good data Sometimes it can detect when a tripod has tipped over or comes out of the water Goodends is often fooled When in doubt, select �all data� Bad data can be trimmed later by using runmask and/or trimfix

34. USGS Apr. 6, 2008 What did that just do? Ensembles at ends of the data file were trimmed using: Deployment and recovery dates Tilt sensor

35. USGS Apr. 6, 2008 Range to Boundary Estimate A range to boundary estimate is always calculated and provided in the output file for uplooking and downlooking cases

36. USGS Apr. 6, 2008 Methods to trim the bins(uplooking) Pressure sensor Most accurate if instrument has one Make sure that it worked USGS surface detect RDI Surface.exe � being phased out User input If you must If the ADCP is equipped with a pressure sensor, that is the most preferable source to obtain depth information, but it is possible for a pressure sensor failure, such as coated with antifouling paint, barnacle growth, or improper zeroing of the sensor. It is important to recognize this as it will affect the outcome of the bin trimming process. If you don�t recognize a pressure sensor failure before processing, you most likely will after trimming the bins because either a lot of bins will be trimmed, or none of them will. Still, however, there can be situations in which, at first glance, the trimming looks accurate, but in reality, it is not. Surface.exe is a program obtained through RDI that looks at the echo intensity of each bin for each ensemble to determine the locations of the water surface. If the ADCP is equipped with a pressure sensor, that is the most preferable source to obtain depth information, but it is possible for a pressure sensor failure, such as coated with antifouling paint, barnacle growth, or improper zeroing of the sensor. It is important to recognize this as it will affect the outcome of the bin trimming process. If you don�t recognize a pressure sensor failure before processing, you most likely will after trimming the bins because either a lot of bins will be trimmed, or none of them will. Still, however, there can be situations in which, at first glance, the trimming looks accurate, but in reality, it is not. Surface.exe is a program obtained through RDI that looks at the echo intensity of each bin for each ensemble to determine the locations of the water surface.

37. USGS Apr. 6, 2008 Select method to trim the bins Pressure is offered as an option if a pressure sensor is present pressure is the best option, followed by USGS surface program A range to boundary estimate is always calculated and provided in the output file for uplooking and downlooking cases

38. USGS Apr. 6, 2008 How Close to the Surface? You may choose how close to the surface � or beyond - to trim the bins Express trim as percentage of water column, i.e. 105 = 105% or full water column plus overshoot by 5% Check the command window for tidal range and mean sea level estimates based on range to boundary information

39. USGS Apr. 6, 2008 Conclusion of Automated Editing This concludes the automated conversion, masking, trimming and editing process. Automation, however, does not always catch all bad data, so additional masking may be performed by hand, which will be discussed next

40. USGS Apr. 6, 2008 Display Data with Ncbrowse Select the data file Select a variable Click to plot the variable

41. USGS Apr. 6, 2008 Check Data with Ncbrowse

42. USGS Apr. 6, 2008 Using Runmask Runmask was used by runadcp to perform automatic masking. We will use it again now to mask by hand. Runmask first made a .msk file to go with the first raw data file translated from the TRDI binary, *.cdf. We will repeat the procedure now, making a second .msk file to go with our final output file, *T.cdf We will then interactively edit the .msk file to add more data to be removed (the data after the tripod tipped over) The final step will apply the mask file and create a new output file with all bad data removed.

43. USGS Apr. 6, 2008 Starting Runmask Invoke runmask at the command line using the final output file You will be prompted for mask file and masked output file names

44. USGS Apr. 6, 2008 Naming the Masked Output Data Since this second masked output file is based on the *T.cdf file, it will be called *TM.cdf This will be the new final result It is not good practice to overwrite output files The mask file *T.msk will automatically be created

45. USGS Apr. 6, 2008 maskADCP Rumask is suspended while maskADCP is open, but any key hit with the MATLAB command window in focus will start the application of the mask file before you may be finished� so be careful!

46. USGS Apr. 6, 2008 maskADCP Controls

47. USGS Apr. 6, 2008 View Other Data Viewing orientation variables to confirm that bad data is due to tipped tripod

48. USGS Apr. 6, 2008 How and When to Mask Can hand mask at your discretion by simply dragging or clicking with the mouse Data quality check for anything that looks like it doesn�t belong Boat, fish, lobster trap, line, etc. Something taking up residence on the transducer Instrument hiccup Tipped deployment platform If you mask too much, you can easily unmask my selecting the data points again.If you mask too much, you can easily unmask my selecting the data points again.

49. USGS Apr. 6, 2008 Masking with maskADCP

50. USGS Apr. 6, 2008 Masking with maskADCP Data showing masked areas Close this window when done masking or changes will not be applied

51. USGS Apr. 6, 2008 Finishing Masking Runmask is waiting for you to finish interactive masking hit enter at the >> orompt to finish up Runmask now applies the mask file *.msk to the data file *T.cdf and outputs a new masked file, *TM.cdf.

52. USGS Apr. 6, 2008 View edited file in Ncbrowse

53. USGS Apr. 6, 2008 Processing files you should now have: 775WH.mat (MATLAB diary file) 775WH000.cdf (netCDF of raw file) 775WH000.msk (mask file, auto masking) 775WH000M.cdf (new ADCP file after masking) 775WH000T.cdf (trimmed file) 775WH000T.msk (mask file, hand masking) 775WH000TM.cdf (trimmed, masked file file)

54. USGS Apr. 6, 2008 Use trimfix.m to trim the end of the data file�again Now we will get back to trimming this file correctly.Now we will get back to trimming this file correctly.

55. USGS Apr. 6, 2008 Find the end of the data Ncbrowse zoomed in on pitch until we can determine the last good ensemble is #5410

56. USGS Apr. 6, 2008 Trimfix.m inputs and outputs

57. USGS Apr. 6, 2008 New file created

58. USGS Apr. 6, 2008 Update the data processing checklist!!!!Update the data processing checklist!!!!

59. USGS Apr. 6, 2008 Use Ncbrowse to check

60. USGS Apr. 6, 2008 Step 2 � adcp2ep.m Convert the trim file to EPIC netCDF format Find that ADCP processing checklist�

61. USGS Apr. 6, 2008 Start adcp2ep.m

62. USGS Apr. 6, 2008 Choose the final output file

63. USGS Apr. 6, 2008 Save the *.nc file as�

64. USGS Apr. 6, 2008 Update the data processing checklist!!!Update the data processing checklist!!!

65. USGS Apr. 6, 2008 Select *.dlg file Wait! We are going to have a little race here to see which computer and operating system is the quickest. Everyone press �Open� on three. 1��2��3.Wait! We are going to have a little race here to see which computer and operating system is the quickest. Everyone press �Open� on three. 1��2��3.

66. USGS Apr. 6, 2008 Wait�.

67. USGS Apr. 6, 2008 Insert appropriate metadata

68. USGS Apr. 6, 2008 You should see this�

69. USGS Apr. 6, 2008 Turn diary off Probably forgot that you were logging all of this didn�t you? Then I guess it�s a good thing this file remembers what processing was performed.Probably forgot that you were logging all of this didn�t you? Then I guess it�s a good thing this file remembers what processing was performed.

70. USGS Apr. 6, 2008 Data Quality Checks ncdump, ncbrowse Check variable minimum,maximum Check attributes maskADCP may also be used as a browser

71. USGS Apr. 6, 2008 NC files in Ncbrowse When viewing *.nc files in ncbrowse, using reverse for depth will plot uplooking ADCP data right side up

72. USGS Apr. 6, 2008 Check quality

73. USGS Apr. 6, 2008 ncdump

74. USGS Apr. 6, 2008 ncdump output

75. USGS Apr. 6, 2008 Double check each entry Water depth�does it agree with the mooring log? Lat/lon�were they typed in correctly? Error velocity�is it too high? Start/stop times�do they make sense? Project/Experiment names?

76. USGS Apr. 6, 2008 That�s it! You have now converted a raw ADCP data file into an EPIC compliant netCDF data file. Questions?????