Ποιότητα και Αποθήκες εδοµένων: Ερευνητικά Θέµατα και Εµπειρίες από το έργο DWQ

Transcript

1 Ποιότητα και Αποθήκες εδοµένων: Ερευνητικά Θέµατα και Εµπειρίες από το έργο DWQ Ιωάννης Βασιλείου Εθνικό Μετσόβιο Πολυτεχνείο 10 Μαΐου

2 OUTLINE Introduction Motivation The Data Warehouse Metadata Framework Developed Architecture, Processes, Quality Models Employing the Framework Conclusions 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 2

3 Foundations of Data Warehouse Quality- DWQ Project National Technical University of Athens (NTUA) (COORDINATOR) Informatik V & Lehr- und Forschungsgebiet Theoretische Informatik (RWTH-Aachen) Institute National de Recherche en Informatique et en Automatique (INRIA) Deutsche Forschungszentrum fόr kόnstliche Intellige (DFKI) University of Rome «La Sapienza» (Uniroma) Istituto per la Ricerca Scientifica e Tecnologica (IRS University of Manchester (UMan) 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 3

4 Introduction Motivation DWQ in a nutshell Contribute to the systematic understanding and usage of the interplay between QUALITY FACTORS and DESIGN / EVOLUTION OPTIONS in Data Warehousing (Objective) Develop comprehensive DW Foundations (Framework), Prototype and Evaluate them (Achievements) Enriched Meta data management facilities (meta-models for architecture, quality, process and evolution) with embedded analysis and optimization techniques (Key Methodology) 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 4

5 Starting Point: Traditional DW Architecture Clients GIS OLAP DSS Examples: Microsoft Repository Metadata Interchange Specification (MDIS) control and manage metadata for OLAP databases. Meta DataBase Repository Data Mart Data Warehouse Mediator Data Mart Administration Agent Wrappers/ / Loaders Sources Text File DB External data 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 5

6 Some Statistics for the Importance of Data Quality for Datawarehouses In a study conducted in 2000 by The Data Warehousing Institute, 93 percent of the respondents recognize the importance of having a data quality strategy for their data warehousing initiatives. 24 percent of the respondents recognize the importance of having a data quality strategy but do not have a plan for implementation. Only 17 percent of the respondents have implemented a data quality strategy, while 30 percent have a plan but have not implemented it, and 22 percent have begun their implementation. 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 6

7 Available Technology - Tools Data quality technologies have been developed that assist organizations with data quality tools and help data warehousing managers ensure that they are loading accurate data into a data warehouse. Data discovery tools help identify errors and inconsistencies in source data. Data conditioning tools help fix those errors by parsing, transforming, matching, and consolidating data elements. Some tools also validate, for example name and address elements against third party databases, such as the U.S. Postal Service database. The result is a single file that is ready for data transformation. 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 7

8 Some Extraction Transformation Loading (ETL) Tools dfpower 3.3 DataFlux Ardent Quality Manager from Informix, Version 4.7 Informix Software, Inc. Innovative Systems Data Quality Solution Innovative Systems, Inc. Innovative Systems Customer Linking Solution Innovative Systems, Inc. Pure Integrate Oracle Corporation INTEGRITY Solutions Vality Technology Incorporated 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 8

9 Typical Configuration for an ETL Tool 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 9

10 Example Migration Cleansing Transformation Loading (refresh) Prevent Reconciliation Detect 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 10

11 Data Migration - Decisions Content of Table Source System Reference Data: Required for reporting purposes Not required for reporting purposes Migrate Data X Do Not Migrate Data X Source System Transaction Data: Required for reporting purposes Not required for reporting purposes X X RDBMS System Data X Source System Application Meta Data or Temporary Processing Data X 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 11

12 Before Cleansing - Example FIRST_NAME LAST_NAME COMPANY_NAME AREA_CODE PHONE STATE sam Adams boston beer co MA Sam Adams Boston beer co, MA Samuel Adams Boston Beer Co SAMUEL ADAMS BOSTON BEER MA Martin Zweig Zweig Funds NY 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 12

13 After Cleansing - Example FIRST_NAME LAST_NAME COMPANY_NAME AREA_CODE PHONE STATE Samuel Adams Boston Beer Co MA Samuel Adams Boston Beer Co MA Samuel Adams Boston Beer Co MA Samuel Adams Boston Beer Co MA Martin Zweig Zweig Funds NY 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 13

14 Transformation Rules 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 14

15 Loading the Data Full Refresh The full refresh loading approach can prevent unwanted data from entering the data warehouse by containing conditions in the load statements Incremental Refresh The incremental refresh loading approach can prevent unwanted data from entering the data warehouse by containing: 1) conditions in the load statements and, 2) unique indexes on the target tables within the data warehouse 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 15

16 Reconciliation Data 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 16

17 DWQ Objective: Going beyond ETL Tools Not only prevention controls Interested in QUALITY of DW Architecture (data objects) DW Processes DW Services Quality is by nature OBJECTIVE, yet the side-effects of bad quality are REAL and very SUBJECTIVE 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 17

18 Industrial Examples: Traditional DW Architecture Example: Telecom Italia need to clean and integrate hundreds of legacy databases with doubtful semantics to enable data mining and new services DWQ Goal: understand quality effects, methods & tools to improve the quality DWQ Solution: description logic support to analyze relationships 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 18

19 Industrial Examples: Team4 Sales Force Automation View Update Sources (legacy systems) Extraction Relational Data Warehouse View Maintenance [SJ96, SJ98] Lotus Notes Server Replication [NJ98] availability and timeliness of client systems availability and data quality of legacy sources maintainability of the software system ==> used DWQ results on update propagation in Team4 Transfer product Lotus Notes Clients (external) 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 19

20 Additional Industrial Applications Ericsson Eurolabs information server data marts on millions of mobile phones: conceptual and physical issues Deutsche Bank/GMI daily refreshed full accounting from 200 databases down to level of individual workplace (1.4 TB annually) metadata bring refreshment workflow from 73 to 4 hours joint project on client-side distribution via web 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 20

21 A Small Motivating Example (Πάνος Βασιλειάδης) MINISTRY of HEALTH (Greece) Data Warehouse: Sources = COBOL files for all the medical centers in Greece (~2400) Transformation and Cleaning Tasks Quality requirements (Goals) «Achieve 100% completeness and consistency of data» 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 21

22 Running Example Detection of the reasons for the inconsistencies between DW data and source (legacy system) data Client tools COBOL file Buffer table Normalized table COBOL file Buffer table Normalized table Materialized view DW 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 22

23 Starting Point: Traditional DW Architecture Clients GIS OLAP DSS PRACTICAL QUESTIONS not Handled in the Traditional Architecture: -- How come the information from the DW is not the same to the one coming from sources? -- What is the effort required to get in the DW information not currently available? -- If I want 100 % correct data in my DW, how do I design it? how often do I refresh it? / Meta DataBase Repository Wrappers/ Loaders Data Mart Data Warehouse Mediator Data Mart Administration Agent --. Sources Text File DB External data 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 23

24 DWQ DW Architecture Clients GIS OLAP DSS query optimiser subsumption reasoner quality manager DWQ Repository Concept Base Data Mart Data Warehouse Data Mart Administration Agent aggregation reasoner Mediator freshness agent Wrappers / Loaders Sources Text File DB External data 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 24

25 Metadata Framework Introduction Motivation The Data Warehouse Metadata Framework Developed Architecture, Processes, Quality Models Employing the Framework Conclusions 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 25

26 Viewpoints of a DW 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 26

27 The DW Traditional Architecture To handle the Questions of Quality, we need to model this Architecture by Introducing the Business (Enterprise) and the Physical Perspectives Quality Issues DW Designer End User Information Directory Meta Schema Business & Technical Metadata User Schema Derived Data Aggregation & Customization Data Warehouse Schema Reconciled Data Extraction & Integration Source administrator OLTP system Source Schema Operational Data 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 27

28 Architecture Meta Model: Enterprise Version (Meta level) Extending the Traditional DW Conceptual Perspective Logical Perspective Physical Perspective Client Model OLAP Client Schema Client Data Store? Aggregation/ Customization Transportation Agent Enterprise Model DW Schema DW Data Store Observation Wrapper Transportation Agent Operational Department Model OLTP Source Schema Source Data Store 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 28

29 Architecture Model Instantiation Levels Conceptual Perspective Logical Perspective Physical Perspective Meta Model Level Client Level DW Level Source Level Models/ Meta Data Level in in in Real World 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 29

30 The Repository for meta models Using TELOS (Mylopoulos et al) Extensible meta-modeling language Both a graphical and a frame syntax Mapped to an underlying formal semantics based on deductive databases All concepts are first-class citizens ConceptBase implementation of TELOS (Aachen) 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 30

31 Architecture Model: Structure of the Meta Model as implemented in ConceptBase / Telos in Measurable Object in in ConceptualObject LogicalObject PhysicalObject isa isa deliversto isa isa isa DW_ Component hasstructure Model hasconcept hastype Schema isa Concept Type isa isviewon Agent DataStore Client Model Enterprise Model Source Model issubsumedby relatesto Client Schema DW Schema isa Source Schema isa Control Agent Transport Agent Client DataStore isa DW DataStore Source DataStore 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 31

32 Process Meta Model: Capturing the Dynamic Aspects of the Architecture Model Conceptual Perspective Logical Perspective Physical Perspective Meta Model Level Client Level DW Level Source Level Process Process Meta Model Models/ Meta Data Level in uses Process Model in in Processes Real World 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 32

33 DW Process Meta Model Workflow Reference Model (made less abstract to fit in the DW case, e.g.: capture schedules, relationships with data) Strategic Dependency Model (adding the conceptual perspective) Processes: Cleaning, transformation, transfer, computation ROLE ACTIVITY AGENT 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 33

34 WHY WHAT HOW: Role Activity - Agent Conceptual Perspective Output Concept Logical Perspective Output Type Physical Perspective Output DataStore Role Activity Agent Input Concept Input Type Input DataStore 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 34

35 Process Meta Model: DW Operational Process Meta Model STAKEHOLDER FOR PERSON MAPPED NEXT IN SCHEDULE DW_USER type active COMPOSITE ACTIVITY COMPOSED PROCESS ELEMENT OPERATES IS RELATED passive ROLE MAPPED isa ACTIVITY isa TRANSITION ELEMENT EXECUTED BY AGENT (application) DW_OBJECT INPUT OUTPUT responsibility isa DATA PACKAGE INPUT/ OUTPUT RELATES TO CONCEPT MAPPED TYPE STORED DATA STORE Conceptual Perspective (why) Logical Perspective (what) Physical Perspective (how) 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 35

36 Quality Model Quality in a Data Warehouse Quality of Data Quality of Processes Quality of Service At all perspectives Establishment of Quality aspects (dimensions) Scientific vs. Pragmatic (user defined) 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 36

37 All Perspectives / Models Metadata Framework Meta Model Level Client Level DW Level Conceptual Perspective Logical Perspective Physical Perspective Process Process Meta Model Quality Metamodel Source Level Models/ Meta Data Level in uses Process Model Quality Model in in Quality Measurements Real World Processes 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 37

38 Quality Model Concepts: Measurable Object (e.g. logical schema of source) Quality Goal (e.g.,improve availability of source A) Quality Query (decide whether a quality goal is achieved) Quality Dimension (e.g., availability, correct ) Quality Factor (measurement) property of a measurable object Stakeholders (decision makers, designers, administrators, programmers) 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 38

39 Quality Dimensions Example: Data Usage Data usage quality accessibility usefulness System availability interpretability timeliness responsiveness Transactional availability security currency volatility 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 39

40 Quality Factors by Perspective Conceptual Perspective Completeness Redundancy Consistency Correctness Traceability of Concepts and Models Logical Perspective Usefulness of schemas Correctness of mappings Interpretability of schemas Physical Perspective Efficiency Interpretability of schemas Timeliness of stored data Maintainability/ Usability of software components - Questions and metrics for each quality factor? - Predictive models of quality impacts and trade-offs? - Can the results be mapped back into data warehouse practice? 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 40

41 Questions and Metrics for Quality Factors refined quality factors for six key DW processes conceptual source integration data integration multidimensional aggregate modeling logical/physical design refreshment querying 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 41

42 Quality Factors Metrics (Schema) Factor Methods of measurement Metrics Schema quality Correctness final inspection of data warehouse schema for each entity and its corresponding ones in the sources number of errors in the mapping of the entities Completeness final inspection of data warehouse schema for useful entities in the sources, not represented in the data warehouse schema number of useful entities, not present in the data warehouse Minimality final inspection of data warehouse schema for undesired redundant information number of undesired entities in the data warehouse trace ability final inspection of data warehouse schema for inability to cover user requirements number of requirements not covered 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 42

43 Quality Factors Metrics (Data Usage) Data Usage Quality Logical Perspective Physical Perspective Schema Type Agent Data Store Accessibility Is the schema definition accessible by the users? Is the type visible and accessible for users? Is the network sufficient for delivered data? Is the data store accessible? Availability Frequency of updates Frequency of updates Response time Uptime of data store, response time Security Level of security (access rights) Level of security (access rights) Are there physical access restrictions? Is the store able to prevent unauthorized access? Usefulness Is the schema used by any users? Is the type used by any users? Is the data delivered by the agent really used in the destination store? Is the data in this store queried by a user? Interpretability Is the schema understandable? Is the type understandable? Is the data delivered understandable? Is the data stored understandable? 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 43

44 Quality Factors Metrics (Data Quality) Data Quality Physical Perspective Agent Data Store Completeness Number of tuples delivered wrt. expected number Number of stored null values where there are not expected Credibility Believability in the process that delivers the Number of tuples with default values values Accuracy Number of delivered accurate tuples Level of preciseness; Number of accurate tuples Consistency Is the delivered data consistent with other data Number of tuples violating constraints, number of coding differences Data Interpretability Number of tuples with interpretable data, documentation for key values, is the format understandable? Number of tuples with interpretable data, documentation for key values, is the format understandable? 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 44

45 Quality Meta Model Management An Adapted GQM Approach DW Designers establish Decision Maker DW Administrator defined on Quality Goal evidence for DW Objects, Processes and Data Metadata for DW Architecture, Quality and Processes Quality Query evaluated by Measurement Processes Quality Factor [Jarke et al., IS 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 45

46 The DWQ Quality Meta Model in ConceptBase Purpose direction description String imposedon concreteby! QualityGoal forp e rs on dimension Stakeholder prefers? QualityQuery Quality Dimension dependson evaluates dimension issubdimof Me a s ure a ble Object onobject Measuring Agent Quality Factor when Timestamp expected achieved unit Me tricunit PowerSet of QualityDomain hasresult 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 46

47 DW Process Quality - Example Role Quality Goal why? Conceptual Achieve 100% consistency of the information to be given to the minister! Activity Quality Query what? Logical Is the propagation activity performing properly? Agent Quality Factor how? Physical Correctness of software processes 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 47

48 Analysis of Quality Factors in the Example Quality Dimens ion DW objects Primary Quality Factors Derived Quality Factors Design Choices Consist ency - COBOL Source file - Buffer table - Normalized table - Materialized View - Loading process - Cleaning process - Computation process - Consistency of a data store - Completeness of a data store - Correctness of an application - Consistency of a data store - Completeness of a data store - Data flow - Chosen source files 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 48

49 Metadata Management Introduction Motivation The Data Warehouse Metadata Framework Developed Architecture, Processes, Quality Models Employing the Framework Conclusions 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 49

50 Employing the Framework -Mapping the Architecture and Models to a Traditional DW Clients GIS OLAP DSS query optimiser subsumption reasoner quality manager aggregation reasoner DWQ Repository Concept Base Data Mart Data Warehouse Mediator Data Mart Administration Agent freshness agent Wrappers / Loaders Sources Text File DB External data 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 50

51 Methodological Approach for Quality Management Quality Goal 4. Re-evaluation & evolution 1. Design 2. Evaluation 3. Analysis & Improvement Define Quality Factor Types Decompose complex objects and iterate Define Object Types Define Object Instances & Properties Empirically derive "functions" Define Metrics & Agents Feed values to quality scenario and play! Discover/Refine new/old "functions" Negotiate! Analytically derive "functions" Produce expected/ acceptable values Take actions! Produce a scenario for a goal Compute! Acquire values for quality factors (current status) 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 51

52 Evolution in the DW Repository Evolution Operators document changes in the DW, and link them to Quality Factors, manually, via analytical models, simulations, or measurements Example: View Materialization DW Completeness affects Add DW Relation next composed Of DataStore Availability DataStore Minimality Evaluate View next Store View Extent Materialize View workson... DW Interpretability next Store View Meta Data Source DataStore DW DataStore [Quix, DMDW 99] 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 52

53 Conclusions Summarizing the Approach Formal Results Tools Developed 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 53

54 Summarizing the DWQ Approach Architecture, Processes, Quality METAMODELS PROCESS METAMODEL QUALITY METAMODEL ARCHITECTURE METAMODEL PROCESS MODELS QUALITY MODEL DATA MODELS MODELS Conceptual Logical Physical Conceptual Logical Physical Clie Enterpris Source Clien Enterprise Source 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 54

55 Key Formal Results on Quality Impacts conceptual: description logic theory and tools for complete reasoning about the relationships between source, enterprise, and client models (Rome, Manchester, Aachen) conceptual/logical: containment, satisfiability, and rewriting of queries over views with & without aggregates (DFKI, Rome) logical/physical: incremental cost-based optimization of view materializations (Athens) physical: detailed impact analysis of replication and refreshment policies (Aachen, INRIA) 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 55

56 Mapping DWQ Back to Practice Decision Maker Metadata Repository R3 R3 R3 Data Marts Aggregations DW Designers Metadata for DW Architecture DW Quality DW Processes DW Evolution Materialized Views R3 Wrappers/ Mediators DW Administrat Sources R3 R3 R3 R3 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 56

57 DWQ Methodology: Sources Metadata Repository 1. Conceptual Enterprise Model 2. Conceptual Source Models R3 S1 R3 Enterprise Model R3 S2 R3 S3 Quality aspects: - semantic source quality - coverage of sources - incrementality Contributions (Lenzerini): - powerful logic reasoners delivered as ER extensions with simple constraints - extensions to non-relational sources started R3 Sn OLTP updates 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 57

58 DWQ Methodology: Client Mappings Metadata Repository C1 C2 Cm User queries 4. Translate aggregates into OLAP operations 3. Conceptual Client Modeling 1. Conceptual Enterprise Model 2. Conceptual Source Models R3 S1 Conj. Queries R3 Enterprise Model R3 S2 Conj. Queries R3 S3 Conj. Queries R3 Sn Quality aspects: - understandability - client flexibility Contributions (Franconi - fully compatible EER with aggregates - fully compatible cube algebra - flexible UI tools OLTP updates 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 58

59 DWQ Methodology: Physical Metadata Repository Quality aspects: - DW timeliness - update+query cost - space restrictions Contributions (Sellis): - optimal materialization - linked with conceptual C1 C2 Cm R3 S1 Conj. Queries Materialized Views R3 Enterprise Model R3 S2 Conj. Queries R3 S3 Conj. Queries User queries 5. Design Optimization R3 Sn OLTP updates 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 59

60 DWQ Methodology : Logical Metadata Repository 1. Conceptual Enterprise Model C1 C2 Cm Conj. Queries Conj. Queries Materialized Views R3 Enterprise Model Conj. Queries User queries Quality aspects: - consistency Contribution (Lenzerini - semi-automatic query generation from design 5. Design Optimization 6. Data Reconciliation 2. Conceptual Source Models R3 S1 R3 S2 R3 S3 R3 Sn OLTP updates 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 60

61 DWQ Methodology: Operations Metadata Repository Quality aspects: - query efficiency - OLTP availability vs. DW timeliness - time/space constraints C1 C2 Cm Conj. Queries Contributions (Nutt/Simon): - algorithms - performance studies R3 S1 Materialized Views R3 Enterprise Model R3 S2 Conj. Queries R3 S3 Conj. Queries User queries R3 Sn 4. Translate aggregates into OLAP operations Rewriting of Aggregate Queries 6. Data Reconciliation Refreshment OLTP updates 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ ιαφάνεια 61

62 Tools Developed Metadata Repository C1 C2 Cm User queries 4. Translate aggregates into OLAP operations 3. Conceptual Client Modeling 1. Conceptual Enterprise Model Conj. Queries Conj. Queries Materialized Views R3 Enterprise Model Conj. Queries Rewriting of Aggregate Queries 5. Design Optimization 6. Data Reconciliation 2. Conceptual Source Models R3 S1 R3 S2 R3 S3 R3 Sn Refreshment 2 Ιουνίου 2001 Ιωάννης Βασιλείου - ΕΜΠ OLTP updates ιαφάνεια 62