Diftong [Elektronisk resurs] a tool for validating big data workflows
-
Rizk, Raya (författare)
-
McKeever, Steve, 1969- (författare)
-
Petrini, Johan (författare)
-
Zeitler, Erik (författare)
-
Uppsala universitet Humanistisk-samhällsvetenskapliga vetenskapsområdet (utgivare)
- Publicerad: 2019
- Engelska.
-
Ingår i: Journal of Big Data. - 2196-1115. ; 6:1
-
Läs hela texten
-
Läs hela texten
-
Läs hela texten
Sammanfattning
Ämnesord
Stäng
- Data validation is about verifying the correctness of data. When organisations update and refine their data transformations to meet evolving requirements, it is imperative to ensure that the new version of a workflow still produces the correct output. We motivate the need for workflows and describe the implementation of a validation tool called Diftong. This tool compares two tabular databases resulting from different versions of a workflow to detect and prevent potential unwanted alterations. Row-based and column-based statistics are used to quantify the results of the database comparison. Diftong was shown to provide accurate results in test scenarios, bringing benefits to companies that need to validate the outputs of their workflows. By automating this process, the risk of human error is also eliminated. Compared to the more labour-intensive manual alternative, it has the added benefit of improved turnaround time for the validation process. Together this allows for a more agile way of updating data transformation workflows.
Ämnesord
- Social Sciences (hsv)
- Media and Communications (hsv)
- Information Studies (hsv)
- Samhällsvetenskap (hsv)
- Medie- och kommunikationsvetenskap (hsv)
- Biblioteks- och informationsvetenskap (hsv)
- Information Systems (uu)
- Informationssystem (uu)
Genre
- government publication (marcgt)
Indexterm och SAB-rubrik
- Big data
- Data testing
- Data validation
- Data quality
- Big data validation process
- Big data validation tool
- Big data workflow
Inställningar
Hjälp
Ingår i annan publikation. Gå till titeln
Journal of Big Data