Auditing Your Data and Answering the Question - Is It the End of the Day Yet?
Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only! Grab it Explore the intricacies of data auditing and workflow optimization in this 37-minute conference talk from NDC Conferences. Dive into Nielsen's robust Kafka architecture and ETL processes, uncovering strategies to track, analyze, and store auditing information. Learn about implementing an AVRO Audit header, designing metadata for auditing heartbeats, and optimizing auditing tables. Discover how to create an alert-based monitoring system using technologies like Kafka, Avro, Spark, Lambda functions, and complex SQL queries. Gain insights into managing partitions with Apache Airflow, offloading data to history, and scheduling Spark jobs. Understand how to detect duplications and implement an effective alerting system. Finally, tackle the age-old question: "Is it the end of the day yet?" through the lens of data processing and legacy problem-solving.