1 / 10

Customer Data Repository

LIVE Workflow. Customer Data Repository . Interesting sequences of event. FTP. LIVE Raw Data Repository. LBIS. LFIS. LIVE Funnel. Transformation Validation. Sequencing Validation. Summarization Validation. LIVE Front-end Input Schema. LIVE Back-end Input Schema. LIVE Stats.

nanji
Télécharger la présentation

Customer Data Repository

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. LIVE Workflow Customer Data Repository Interesting sequences of event FTP LIVE Raw Data Repository LBIS LFIS LIVE Funnel Transformation Validation Sequencing Validation Summarization Validation LIVE Front-end Input Schema LIVE Back-end Input Schema LIVE Stats transformation StatisticalAnalysis LUIS Report Generation and Visualization LIVE UI Schema

  2. LIVE Workflow Interesting sequences of event Interesting sequences of event shell & HIVE scripts Customer Data Repository Interesting sequences of event FTP LIVE Raw Data Repository LBIS LFIS LIVE Funnel Transformation Validation Sequencing Validation Summarization Validation LIVE Front-end Input Schema LIVE Back-end Input Schema LIVE Stats transformation StatisticalAnalysis LUIS Report Generation and Visualization LIVE UI Schema

  3. LIVE Raw Data Repository Customer Data Repository SCP hadoopfs -put FTP LIVE Raw Data Repository LBIS LFIS LIVE Funnel Transformation Validation Sequencing Validation Raw-customer tables and maptables Summarization Validation GUID propagation Cleaning & truncation LIVE Stats transformation StatisticalAnalysis Joining with maptables LBIS Transform in LBIS LUIS Report Generation and Visualization

  4. LIVE Raw Data Repository Customer Data Repository SCP hadoopfs -put FTP LIVE Raw Data Repository LBIS LFIS LIVE Funnel Transformation Validation load_local.sh load_hdfs.sh hive -f stage.sql hive -f lbis.sql Sequencing Validation Raw-customer tables and maptables Summarization Validation GUID propagation Cleaning & truncation LIVE Stats transformation StatisticalAnalysis Joining with maptables Transform in LBIS LUIS Report Generation and Visualization LBIS

  5. LIVE Raw Data Repository Customer Data Repository SCP hadoopfs -put FTP load_local.sh load_hdfs.sh hive -f stage.sql hive -f lbis.sql LIVE Raw Data Repository LBIS LFIS LIVE Funnel .................................... .................................... .................................... .................................... .................................... -- GUID Propagation CREATE TABLE userid_guid_mapAS SELECT distinct user_id, guid from dbact_guid; CREATE TABLE mapcountAS SELECT COUNT(*) AS cnt, user_id FROM userid_guid_mapGROUP BY user_id; CREATE TABLE impr_guidAS SELECTa.*, m.guidFROM impri JOIN userid_guid_mapm ON (i.user_id=m.user_id); SELECT COUNT(*) FROM impr_guid; .................................... .................................... .................................... .................................... .................................... .................................... Transformation Validation Sequencing Validation Raw-customer tables and maptables Summarization Validation GUID propagation Cleaning & truncation LIVE Stats transformation StatisticalAnalysis Joining with maptables Transform in LBIS LUIS Report Generation and Visualization LBIS

  6. LIVE Raw Data Repository Customer Data Repository SCP hadoopfs -put FTP load_local.sh load_hdfs.sh hive -f stage.sql hive -f lbis.sql LIVE Raw Data Repository LBIS LFIS LIVE Funnel .................................... .................................... .................................... .................................... .................................... -- GUID Propagation CREATE TABLE userid_guid_mapAS SELECT distinct user_id, guid from dbact_guid; CREATE TABLE mapcountAS SELECT COUNT(*) AS cnt, user_id FROM userid_guid_mapGROUP BY user_id; CREATE TABLE impr_guidAS SELECTa.*, m.guidFROM impri JOIN userid_guid_mapm ON (i.user_id=m.user_id); SELECT COUNT(*) FROM impr_guid; .................................... .................................... .................................... .................................... .................................... .................................... Transformation Validation Sequencing Validation Raw-customer tables and maptables Summarization Validation GUID propagation Cleaning & truncation LIVE Stats transformation StatisticalAnalysis Joining with maptables Transform in LBIS Pre-step LUIS Report Generation and Visualization Stats LBIS Action Validation

  7. LIVE Raw Data Repository Customer Data Repository SCP hadoopfs -put FTP load_local.sh load_hdfs.sh hive -f stage.sql hive -f lbis.sql LIVE Raw Data Repository LBIS LFIS LIVE Funnel .................................... .................................... .................................... .................................... .................................... -- GUID Propagation CREATE TABLE userid_guid_mapAS SELECT distinct user_id, guid from dbact_guid; CREATE TABLE mapcountAS SELECT COUNT(*) AS cnt, user_id FROM userid_guid_mapGROUP BY user_id; CREATE TABLE impr_guidAS SELECTa.*, m.guidFROM impri JOIN userid_guid_mapm ON (i.user_id=m.user_id); SELECT COUNT(*) FROM impr_guid; .................................... .................................... .................................... .................................... .................................... .................................... Transformation Validation Sequencing Validation Raw-customer tables and maptables Summarization Validation GUID propagation Cleaning & truncation LIVE Stats transformation StatisticalAnalysis Joining with maptables Transform in LBIS Pre-step LUIS Report Generation and Visualization Stats LBIS Action Validation

  8. LIVE Raw Data Repository Customer Data Repository SCP hadoopfs -put FTP load_local.sh load_hdfs.sh hive -f stage.sql hive -f lbis.sql LIVE Raw Data Repository LBIS LFIS LIVE Funnel .................................... .................................... .................................... .................................... .................................... -- GUID Propagation CREATE TABLE userid_guid_mapAS SELECT distinct user_id, guid from dbact_guid; CREATE TABLE mapcountAS SELECT COUNT(*) AS cnt, user_id FROM userid_guid_mapGROUP BY user_id; CREATE TABLE impr_guidAS SELECTa.*, m.guidFROM impri JOIN userid_guid_mapm ON (i.user_id=m.user_id); SELECT COUNT(*) FROM impr_guid; .................................... .................................... .................................... .................................... .................................... .................................... Transformation Validation Sequencing Validation Raw-customer tables and maptables Summarization Validation Query failed (MR job not launched) GUID propagation Mapred job failed Hadoop stopped responding Cleaning & truncation LIVE Stats transformation StatisticalAnalysis Joining with maptables Transform in LBIS Pre-step LUIS Report Generation and Visualization Stats LBIS Action Validation

  9. LIVE Raw Data Repository Customer Data Repository SCP hadoopfs -put FTP load_local.sh load_hdfs.sh hive -f stage.sql hive -f lbis.sql LIVE Raw Data Repository LBIS LFIS LIVE Funnel .................................... .................................... .................................... .................................... .................................... -- GUID Propagation CREATE TABLE userid_guid_mapAS SELECT distinct user_id, guid from dbact_guid; CREATE TABLE mapcountAS SELECT COUNT(*) AS cnt, user_id FROM userid_guid_mapGROUP BY user_id; CREATE TABLE impr_guidAS SELECTa.*, m.guidFROM impri JOIN userid_guid_mapm ON (i.user_id=m.user_id); SELECT COUNT(*) FROM impr_guid; .................................... .................................... .................................... .................................... .................................... .................................... Transformation Validation Sequencing Validation Raw-customer tables and maptables Summarization Validation Query failed (MR job not launched) GUID propagation Mapred job failed Hadoop stopped responding Cleaning & truncation LIVE Stats transformation StatisticalAnalysis Joining with maptables Transform in LBIS Pre-step LUIS Report Generation and Visualization Stats LBIS Action --Exception Handling !<cmd> Validation

  10. LIVE Raw Data Repository Customer Data Repository SCP hadoopfs -put FTP provision_cluster.sh load_local.sh load_hdfs.sh hive -f stage.sql hive -f lbis.sql terminate_cluster.sh LIVE Raw Data Repository LBIS LFIS LIVE Funnel .................................... .................................... .................................... .................................... .................................... -- GUID Propagation CREATE TABLE userid_guid_mapAS SELECT distinct user_id, guid from dbact_guid; CREATE TABLE mapcountAS SELECT COUNT(*) AS cnt, user_id FROM userid_guid_mapGROUP BY user_id; CREATE TABLE impr_guidAS SELECTa.*, m.guidFROM impri JOIN userid_guid_mapm ON (i.user_id=m.user_id); SELECT COUNT(*) FROM impr_guid; .................................... .................................... .................................... .................................... .................................... .................................... Transformation Validation Sequencing Validation Raw-customer tables and maptables Summarization Validation Query failed (MR job not launched) GUID propagation Mapredjob timeout on 95% reduce Hadoop stopped responding Cleaning & truncation LIVE Stats transformation StatisticalAnalysis Joining with maptables Transform in LBIS Pre-step LUIS Report Generation and Visualization Stats LBIS Notification Action --Exception Handling !<cmd> Status record Validation Restart

More Related