HopeDale:Hardware-optimized Progressive Data Processing for Exploratory Data Analyses

2021 / 03 / 04

Prof. Dr. Stefan Manegold,Database Architectures research group, Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands

Prof. Shimin Chen (CAS), Institute of Computing Technology, CAS

Creation: 2019

Project Description

Led by CWI, the team studied the requirements and challenges of the progressive data processing paradigm, in particular as they are formed by interactive visualization applications, identified the differences with the existing paradigms of Approximate Query Processing and Online Aggregation and has focused on two relevant directions: (1) the expression and efficient evaluation of progressive queries and (2) the evaluation of progressive queries under specific preferences on subsets of the data provided by the user.

For the first direction, the team is working on extending the syntax of the SQL query language to express modifications among progressive queries, in order to provide a traditional DBMS with information about which earlier results and/or intermediate data structures/steps can be reused for the evaluation of the latest submitted query.

Modifications to SQL queries can be coupled with common user interactions on a visualization interface based on the semantics of these interactions, which can also be used as hints about what types of subsequent queries are anticipated.

The team is in the process of experimenting with techniques inspired by Multi-Query Optimization to evaluate their benefit on executing progressive queries. We are using the latest benchmark data on the area of interactive data exploration as representative examples and use case scenarios for the currently investigated techniques.

The second direction addresses scenarios where the user has preferences over subsets of the data and is interested in viewing results on these subsets early in the query processing. For these scenarios, led by CAS, the team is working on integrating the concept of preference to common data processing algorithms, such as different join algorithms, and implementing techniques for efficiently evaluating these algorithms.

+86 (0) 10 8254 4772

+86 (0) 10 8254 4594

5F, Intelligent Building N.95, Zhong Guan Cun East Road, Beijing, China