Appunti - Technologies for Informative System

Notes on technologies for information system

Intro to big data integration

The main driver for the change of paradigms in data management includes:

• Data science as a discipline that combines elements of data cleaning and transformation, statistical analysis, data visualization, and ML techniques
• Cloud computing has become mainstream
• Data lakes and modern data warehouse query engines
• Interest in data governance in industry

The data integration problem involves combining data from different data sources, providing the user with a unified vision of the data. This includes detecting correspondences between similar concepts from different sources and resolving conflicts. The aim of a data integration system is to set up a system where it is possible to query different data sources as if they were one.

The four V's of big data integration

• Volume: Not only the volume of data, but also the number of data sources
• Velocity: Rate at which data is being collected and data source changes
• Variety: Heterogeneity of data sources, including structure and description
• Veracity: Quality of data in terms of coverage, accuracy, validity, timeliness, ethics, consistency, and completeness

Heterogeneity derives from various forms of autonomy: design, communication, and execution.

The steps of data integration

• Schema reconciliation: Mapping the data structure if it exists. Trying to extract the structure of the data from the data itself
• Record linkage: The process of identifying and linking records that correspond to the same real-world entities in different data sources (e.g., entity resolution, data matching)
• Data fusion: Bringing together relevant information from various sources to create a more complete and accurate database, including reconciliation of non-identical content

Phase of data integration

1. Source schema reverse engineering
2. Related concept identification + conflict analysis and resolution
3. Global conceptual schema design
4. Conceptual to logical translation of the global schema
5. Mapping definition

Ways of integrating database systems

• Materialized database: Data are merged in a new database (warehouse). Periodical ETL operations update the data, useful for analysis and data mining to build a history of the enterprise.
• Virtual non-materialized database: Data remains at the source and is retrieved with a unified interface, always returning a fresh answer to the query. Used when the integrated data needs to be always up-to-date (e.g., virtual multidatabase, global schema).

Large common stores are called Warehouses, and the software used to extract, scrape, transform, and load data into warehouses is called an ETL system. The main goal of the warehouse is to allow systematic or ad-hoc querying. Performing ETL periodically builds up a history of the enterprise.

Schema reconstruction

Data Integration is possible even in a unique database to avoid replicas and redundancy that cause inconsistencies and memory occupation. This occurs typically when different parts of the same enterprise design different data management systems (e.g., ER) but some data is duplicated among the sections.

We study data integration for:

• Federated databases: Homogeneous and heterogeneous data
• Transient, initially unknown data sources

Entity resolution is the problem of accurately identifying multiple, differing, and possibly contradicting representations of unique real-world entities in data.

Designing data integration in the Multidatabase (same data model)

• Source schema identification (when present)
• Source schema reverse engineering
• Conceptual schema integration and restructuring: conflict resolution
• Conceptual to logical translation (global schema)
• Mapping between the global logical schema and the single schemata (logic view definition)

Conflict resolution

• Related concept identification
• Conflict analysis and resolution
• Conceptual schema integration

Types of conflicts

• Name conflict: Same concept, different name
• Type conflict: Different store type
• Data semantic conflict: Same concept, different type of measurement ([euro, dollar], [birthday, age])
• Structure conflicts: Attribute vs entity, same attribute in different entities
• Cardinality conflicts: Different cardinalities in a relationship
• Key conflict: Same concept, different key

Mapping between the global logical schema and single source schemata

Global As View (GAV): The global schema is expressed in terms of data source schemata. Used when data sources are quite stable; however, when the data source changes, the structure of the global view must also change. Schema derives from integrating processes.

• A GAV mapping is a set of assertions, one for each element g of the global Schema G, which implies the mapping of each query g to a query over the data sources.
• Mapping quality depends on how well the source is compiled into the global schema.

In practice, a view is used to express the global schema with unions of sources.

Local As View (LAV): The global schema is independent from the data source. The relationship between sources and the global schema is obtained by defining each source as a view over the global schema, allowing a more flexible global schema that is suitable for stable global schema.

• A Mapping LAV is a set of assertions, one for each element s of each source S.
• Mapping quality depends on how well the sources are characterized.
• Query processing requires reasoning.

In practice, a view with the global schema pattern is created for each source.

GLAV (Global and Local As View): The mapping between sources and the global schema is obtained by defining a set of views, some over the global schema and some over the data sources.

Record linkage and data fusion

Record linkage (also known as entity resolution) is the process of finding information that refers to the same real-world entities. If the fields are distinct, it is much easier to spot similarities with respect to a single string.

String matching

Considering string similarity, to find matches, we can use a similarity function. Different similarity measurements include:

• Sequence-Based: Edit (Levenshtein) distance, based on the minimal number of operations (insert, deletion, replacement) required to transform string a into string b
• Set-Based: Viewing strings as multisets of tokens and using measurements such as the Jaccard Measure, where strings are divided into tokens
• Phonetic: Matching strings based on their sound

Entity matching

To determine when two records are equal, we have different techniques for record similarity:

• Rule-Based: Manually specified rules require time and effort
• Learning Matching Rules: ML supervised or unsupervised techniques (large datasets needed)
• Probabilistic Matching: Models the matching domain using a probability distribution; provides a general framework but is computationally expensive and often hard to understand

Data fusion

Data fusion reconciles inconsistent information once it is recognized that two items refer to the same entity. Inconsistency may result from incorrect or partially viewed sources. Often the correct value can be obtained as a function (data layer).

Data wrappers are components linked to a source that convert queries into queries understandable by a specific data source, translating the result back to the destination format. They must be updated if the source changes. Human-developed wrappers are expensive, leading to the development of automatic techniques. They can extend the query possibilities of a data source but are difficult to use with semi-structured data.

Semi-structured data integration

Semi-structured data refers to data with some structure that is not as prescriptive, regulated, or complete as structured data (e.g., web data, XML). Typically, a semi-structured data model is based on text, trees, or graphs. An overall data representation is progressively built as new information sources are discovered and explored.

MAV and LAV are no longer sufficient; instead, a mediator is required to integrate complex systems.

The term mediation includes:

• The processing needed to make interfaces work
• The knowledge structures that drive the transformation needed to turn data into information
• Any intermediate storage that is needed

The main problem is that each different domain requires a mediator appropriately designed to understand its semantics. The mediator must have access to domain metadata.

Mediators are interfaces specialized in a certain domain placed between the application and the wrappers of the sources. The goal of this interface is to catch the query written in the application domain, decompose it, and forward each piece to the specific wrapper. It also accounts for merging the different responses.