What to Do when Privacy Issues Screw It Up: Ingestion Refactoring in a Big-Data Pipeline

Privacy is an increasingly important concern especially in the European Union (EU). With the growing use of technology individuals’ personal information is being collected and processed by companies and organizations on a massive scale. In order to be compliant with Privacy regulations and General Data Protection Regulation (GDPR) in particular, we could no longer use a software tool, LOGGIT, for several reasons. This tool was a cornerstone in one of our Big-data pipeline ingestion. We did our best to comply with this requirement as soon as possible. In this work, we discuss how we refactor our pipeline architecture several times in order to find a balance between our requirements in terms of reliability and the regulations of the GDPR.