1 / 21

Changes to Data Sources in V10

Changes to Data Sources in V10. KHUG 2009. Peter Heweston. Reassurance. If you leave everything alone, data sources will work just like suffixes and data sources did before Upgrade will look after you. Data Sources Changes - Drivers.

chung
Télécharger la présentation

Changes to Data Sources in V10

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Changes to Data Sources in V10 KHUG 2009 Peter Heweston

  2. Reassurance • If you leave everything alone, data sources will work just like suffixes and data sources did before • Upgrade will look after you

  3. Data Sources Changes - Drivers • Traditionally, the way Hydstra time series data was identified has been very file-specific, particularly for TS files. • Also very arbitrary: “.A” = archive, “.B”..“.Z” for work area files • Other products in the Hydstra / Kisters family use a more tree-oriented view of the data world • TSM, HydroTel, TimeStudio, Wiski • The user decides how the data is arranged • Hydstra’s “switchable datasets” feature • Not an elegant solution • Did not fit Kisters world view • Part of aligning all Kisters products with TSM concepts to allow for future integration

  4. All datasources registered in DATASRC.INI • In Hydstra V9 there were four classes of TS data: • Single-letter TS files (A, B..Z) (suffixes in old parlance) • “Presets” from DATASRC.INI (WQ, GW, TE etc.) • Fully specified data source expressions: • SRC=CONST,VALUE=1.25 • now CONST(1.25) • TS files in other datasets • Usually access via a datasource • Edited by changing dataset • In Hydstra V10 everything comes from DATASRC.INI • No exceptions • However we do offer parameterised datasources

  5. The format of DATASRC.INI has changed • Each data source is a section (rather than a single-line entry) • Type = TSFILE • ExpressionType = NORMAL • Desc = Compatibility-Mode Archive Files • TSClass = ARCHIVE • ULevRead = TS=0 • ULevWrite = TS=2 • Path = &hyd-hydpath. • Extension = A • IndexExt = AX • BakFiles = &hyd-hydpath.backup,3 • ReadonlyFiles = Yes • RecycleBin = No • WBVisible = Yes • WBEditable = No • WBEditInPlace = No • HFShortcuts = No • Blocklimit = 3000 • RegionControl = Yes • Deprecated = No

  6. The format of DATASRC.INI has changed (contd.) • Don’t panic: • The V10 UPGRADE procedure automatically upgrades your existing DATASRC.INI to the new format • However since this is an old-school compatibilityupgrade, it won’t take advantage of most of the new features. • But everything (including your scripts) should continue to run unchanged

  7. DATASRC.INI has changed • Lots of scary-looking keywords • Don’t worry, you should never have to edit DATASRC.INI manually • HYDSWIZ introduced • Special new application to help you manage all your data sources • Provides default valuesand short reminders for all keywords required for each data source type • Provides detailed reports and suggestions if anything is wrong

  8. New concept for TS files: TSCLASS • All TSFILE datasources divided into something called TSCLASSThere are four choices: ARCHIVE – the primary managed archive of TS file dataThere must be one and only one datasource of this class WORK – files intended to become part of the archive in due course AUX – auxiliary data Neither the primary archive, or intended to be merged with it in the future. • Virtual data sources (Perl data sources) • Data from other systems (WQ, GW converted to TS) • Data from other organisations. TEMP – temporary filesFor use in scriptsBelong to none of the categories above, will not appear in the Data Managers’ Workbench.

  9. No Magic Behaviours • TSFILE DataSources now define settings previously either controlled by HYCONFIG.INI, or hardwired into the system • User level required to read this data • User level required to write this data • Create .BAK files? • How many .BAK files to keep? • Mark files as read-only in the folder? • Send deleted files to recycle bin? • Enforce (or relax) region control? • As a result of this, the following keywords are removed from HYCONFIG.INI: • BAKFILE, ROHYD, RecycleArch, RecycleWork, TSArchive, TSWork, TSTemp, WBEditArchive, WBIPEdArchive, WBIPEdWork, HFShortCuts

  10. Free naming of data sources • Data sources can have meaningful names • Including TS file data sources! • HYDSYS01.ARCH instead of HYDSYS01.A • HYDSYS01.TELEM instead of HYDSYS01.T • HYDSYS01.EDIT instead of HYDSYS01.E • Length of data source on parameter screens is 8 • But they can go longer if required • Lets you put more meaning into the data source • COMP_AT suggesting: composite, archive + telemetry

  11. That pesky ol’ SEARCH mode • Not a universally loved feature • Writing TS data to the next “free” data source if the one you specified already exists • We came quite close to ditching it altogether • It is now under your TOTAL control: • If you don’t want it, you don’t have to have it • You can quarantine it to a subset of your work data sources, eg: • TELEM, TELEM.1, TELEM.2, TELEM.3 • Put an end to runaway suffixes • Never stomp on T files again

  12. More control over folders & file extensions • You have complete control over which folders your TS files live in, and their file extensions • Different WORK data sources an live in different folders • Each data source can live in its own folder if you like • HYDPATH and WORKPATH have no guaranteed meaning! • A new HYCONFIG path introduced • TSPATH - \hyd\dat\ts • Each new TS file work area should live in a folder under this • But it is under your control

  13. So long, Suffix • The term “Suffix” is no longer part of Hydstra terminology • Data source = where you get data from • Extension = the thing on the end of a file • You should not see 'suffix' on any parameter screen or report

  14. CONST data sources more flexible • Some data sources can now have parameters • One very useful example of this is the new CONSTdata source • The constant value can be passed in as a parameter:CONST(1.25)CONST(12.51)CONST(-2.075)CONST(42)

  15. Perl Datasources • Perl datasources can be much more powerful • Can have parameters, no need to replicate different versions • T_LAG datasource example • TSCALC datasource • Under construction • Based on TSCALC table in TSADMIN manage • Has a dated period table • Different algorithms at different times • Use any technique (Perl, MODSYN, batch etc) • HYPLOT Demo • Demo Percentile Datasource

  16. Switchable datasets withdrawn • Switchable datasets were a worry • They originally allowed you to override the locations of some DB tables (like RATEPATH) • This became a problem in V10, where ALL DB tables now live in DBFPATH • Many users were using parallel datasource / dataset pairs • Switch to the dataset to manage the data • Use the datasource to access it without switching • Switchable datasets have been withdrawn • We believe that the new features in the datasource system makes up for the loss • In particular, multiple WORK and AUX datasources

  17. Datasets and Databases • Datasets have been withdrawn • All tables live in DBFPATH or SQL • Use work areas for separating tables out • Use DBSEARCH to use separate tables • E.g. • HYDBUTIL copy RATE*.* to [priv.myrate] • Run HYPLOT /e=hyconf=dbsearch;[priv.myrate]

  18. Impact on scripts: “A” for Archive • “A” may not be the archive datasource anymore • We have introduced a new “pseudo-HYCONFIG” variable: ArchiveDataSource • Pseudo-HYCONFIG? • Because it is not stored in HYCONFIG.INI • It is defined by the contents of DATASRC.INI • But you access it as if it were a HYCONFIG setting In perl: my $ArchDS = HyconfigValue('ARCHIVEDATASOURCE'); In DOS batch jobs: call hysetenv ARCHIVEDATASOURCE hyrep %site% %ARCHIVEDATASOURCE% ... INI files can specify it via macro substitution: DataSource = &hyd-ARCHIVEDATASOURCE.

  19. Impact on scripts: folders and file extensions • Directory-listing techniques are no longer reliable: • TS files may not live in HYDPATH or WORKPATH anymore • TS files may not have single-letter file extensions anymore • TS index files may not be “data file plus X” anymore • The new application HYDSQRY should be used instead • List all sites for a given data source (optional site list filter) • List all TSFILE data source (optional site list filter) • List all TSFILE files (optional datasource, site list filter) • List all stray files – files that don’t belong in all TSFILE folders • List all locked filesfiles that being held open by users • Perl wrappers / utilities will be provided

  20. Impact on scripts: site list expressions • Site lists using DIR() may not work anymore • They rely on a folder and filemask • Two new Site List functions have been introduced: • TSFILES(datasrc1,datasrc2…) • List all sites that have TSFILES for the given datasource(s) • TSFILES(ARCH,TELEM) • DSOURCES(tsclass1,tsclass2…) • List all datasources for the given TSCLASS(es) • DSOURCES(ARCHIVE,WORK) • You can combine these two functions: • TSFILES(DSOURCES(ARCHIVE,WORK))

  21. Changes to Data Sources in V10 KHUG 2009 Peter Heweston

More Related