Appunti - System and Method for Big and Unstructured Data

Notes on big and unstructured data

Big data

Four V of Big Data:

• Volume: Not only volume of data, but also number of data sources (data at scale)
• Velocity: Rate at which data is being collected
• Variety: Heterogeneous data sources: structure, description (data in many forms)
• Veracity: Quality of data in terms of coverage, accuracy, validity, timeliness, ethics, consistency

Data scientist goals are:

• Collect
• Analyze
• Build value

Other objectives include obtaining predictable, actionable insights from data, creating products with immediate impact, communicating relevant business stories from data, and building confidence in decisions that drive business value. Before solving a problem, we were exploiting only a subset of the data (the most interesting ones), now we manage to analyze all available data in the same moment.

Toward NoSQL

New paradigm: schema-less idea. This avoids dealing with complex data schema to manage data. We moved from a Schema on Write to a Schema on Read paradigm, improving flexibility. This allows for a single, shared, and fully available BD for the enterprise (data lake).

The process of data is the following:

• Data Ingestion: Importing, transferring, and loading data for storage and later use
• Data Wrangling: Clean raw data and transforming it into data that can be analyzed (understand, cleanse, augment, shape)
• Extract Transform Load (ETL): Used to better fit the targeted query

Transactional properties

A transaction is an elementary unit of work performed by an application. A transactional system (OLTP) is capable of providing transactions. ACID Properties:

• Atomicity: A transaction is an atomic transformation from the initial state to the final one
• Consistency: The transaction satisfies the integrity constraints (initial-final state are consistent)
• Isolation: A transaction is not affected by the behavior of the other concurrent transactions
• Durability: The effect of a transaction that has successfully committed will last forever

CAP theorem

It is impossible for a distributed system to simultaneously provide all 3 of the following guarantees:

• Consistency: All nodes see the same data at the same time
• Availability: Node failures don’t prevent other survivors from continuing to operate
• Network Partition Tolerance: System continues to operate despite arbitrary network partition fail

A distributed system can satisfy any 2 of these guarantees at the same time but not all 3. In networked systems, network partition tolerance is quite necessary.

NoSQL world

• Key Value Store: A key that refers to a payload
• Column Store: Column data is saved together, as opposed to row data
• Document/XML/OBJ Store: Key points to a serialized object
• Graph Store: Nodes are stored independently, and the relations are stored with data

ACID vs BASE properties

ACID properties may not hold with NoSQL DBs. Instead, they guarantee BASE properties:

• Basic Availability: Fulfill requests, even in partial consistency
• Soft State: Abandon the consistency requirements of the ACID Model
• Eventual Consistency: At some point in the future, data will converge to a consistent state

A fully ACID DB is the perfect fit for use cases when reliability and consistency are essential.

Data ingestion: API

The 3 W of API:

• What: An Application Program Interface is a set of routines, protocols, and tools for building software applications, allowing programmatic access to data and platform
• Why: Separating between model and presentation, regulate access to data and avoid direct access to the platform
• Who: Every data provider

How to expose API

1. Each API exposes a set of HTTP(s) endpoints (URLs) to call in order to get data or perform actions on the platform. Most of the endpoints can be “tweaked” via one or more parameters.
2. REST architecture can be used to expose API. RESTful APIs are resource-based WebAPI with a standard URL format. Almost all the APIs require some kind of user authentication. The user must register with the developer of the provider to obtain the keys to access the API.

Some problems

Crawling: Getting a lot of data points from an API: An API Crawler is a software that methodically interacts with a WebAPI to download data or to take some actions at predefined time intervals.

Pagination: Most APIs support data pagination to split huge chunks of data into smaller sets of data.

Timeline: Most databases leverage the concept of a timeline. The solution to retrieve always-new data is a cursoring technique. Instead of reading from the top of the timeline, we read the data relative to the already processed IDs.

Parallelization and multiple accounts: When possible, make parallel requests to gather more data in less time. This can be enabled by handling multiple accounts. Multiple accounts can be managed in different ways:

• Request Based: Round Robin: Strategy among all accounts for the requests
• Account Pull: Sequential account use until the limit is reached
• Account Based: Request Stack: The accounts, in parallel, get the next request from the pool

When APIs are not present, but data retrieval is needed, scraping could be a solution but it should be the last solution.

Graph databases - Neo4J

A Graph is a set of nodes joined by a set of lines or arrows. In computer science, a graph is an abstract data type and could be implemented in different ways:

• Representation Matrix: Incident [ExV], Adjacency [VxV]
• Representative List: Edge or Adjacency List

Graph databases are used when relationships between objects are more important than the objects themselves. They provide an index-free adjacency. Queries are based on a pattern-recognition functionality.

Neo4j

Neo4j is a schemaless DB that maintains the ACID properties, typically employed in non-analytic DB. It offers highly performant read and write capabilities, reliability for mission-critical production, and is not efficient in whole-graph analysis. Relations have direction, but you can navigate them in both directions. In this case, indexing is useful to find an entry point in the graph. Queries are expressed with a custom language called Cypher. The profile operation provides the complete analysis of the selected query.

Hints:

• Use parameters instead of literals
• Set an upper limit to *
• Return only the data you need
• Use Profile/Explain to analyze the performance of your queries

Shortest Path: The shortest path function works with the where condition, it returns the shortest path that respects that following where condition. We have two kinds of conditions:

• Fast Algorithm: Where condition can be checked “online” during the computation of the path
• Exhaustive Algorithm: Where we have to compute the whole path to check the condition (length)

If we use a with operator between the computation of the shortest path and the where condition, the computation of the shortest path will ignore those constraints.

Key value database - Redis

Typically used for performance goals, they are deployed with 3 instances:

• Stand Alone (unusual)
• Integrated with other DB
• For caching purpose

Caching purpose - MemCached

Generic in nature, intended for use in setting up dynamic web applications by alleviating DB load. Technically, it is a pool of servers managed by the client. Data stored in memcache are with high demand, expensive to compute, and common.

Main Features:

• Fast Network Access
• No Persistence or Redundancy/Failover
• No Authentication (security concern)

Redis

Redis is an advanced key-value store. It supports a set of atomic operations and is best used for rapidly changing data with a foreseeable database size. In Redis, values are complex.