GDPR and SOC 2 Compliant Data Pipeline on AWS
AWS Glue · KMS · S3 · IAM · CloudTrail · Macie · Step Functions · PostgreSQL RDS
Client details are anonymized under NDA. Architecture, metrics, and war stories are from the real production system.
- A fintech operating in Germany and the Netherlands was handling GDPR erasure requests with a spreadsheet and email chains. It took 2-3 weeks per request, and a spot check showed roughly 30% of requests missed at least one downstream system.
- I built a centralized pipeline: Glue ETL with field-level KMS encryption, a consent registry in Postgres, and Step Functions orchestrating erasure across all twelve systems with retries and rollback.
- Erasure now completes inside a self-imposed 72-hour SLA, and the system went through a SOC 2 Type II observation period with no exceptions noted. Compliance infra costs roughly $1,200/month.
The Challenge
Constraints
- No existing orchestration platform. The client had no Airflow footprint, and the platform team was two people.
- EU data residency: all customer data had to stay in eu-west-1, enforced technically via SCPs, not by policy document.
- The twelve source systems were owned by different teams. I could not mandate changes to their internal schemas.
- Compliance infrastructure budget was soft-capped around $1,500/month.
- SOC 2 observation window was already scheduled. Roughly five months to be audit-ready.
Key Decisions
Erasure orchestrationStep Functions
Considered: Cron-driven Lambda polling an SQS deletion queue / Managed Airflow (MWAA) / Step Functions
Why: The execution history is itself audit evidence. When the auditor asked "prove this request touched all twelve systems," I handed over the state machine execution log instead of reconstructing it from application logs.
Trade-off: Vendor lock-in, and the state machine definition is now roughly 600 lines of ASL JSON. Not pretty, but defensible.
Encryption granularityPer-field-type keys (names, emails, national IDs)
Considered: Bucket-level SSE-KMS / One data key per record / One customer-managed key per PII field type
Why: Bucket-level SSE protects against lost disks, not against a compromised role that can read the bucket. Per-record keys would have been ideal for crypto-shredding but blew past KMS request limits at our volume.
Trade-off: Erasure still requires Parquet partition rewrites for the data lake. The one decision I would revisit.
Consent registry storePostgreSQL RDS
Considered: DynamoDB / PostgreSQL RDS
Why: Consent state is relational by nature: subjects, consent types, state transitions, legal bases. The compliance team also needed ad-hoc SQL access for regulator queries, which DynamoDB makes painful.
Trade-off: One more instance to patch and fail over. Multi-AZ covered the SOC 2 availability criteria.
PII detectionMacie
Considered: Custom regex scanning in a Lambda / AWS Macie
Why: A custom scanner is a forever-maintenance project. Macie's managed identifiers handled the common cases, and "we use the AWS-native service" is an easier conversation with auditors.
Trade-off: Cost scales with scanned volume, and the false-positive tuning took real effort.
The Approach
GDPR Pipeline Architecture
GDPR Compliance Pipeline Architecture
Field-level encryption, right-to-erasure orchestration, PII detection, and tamper-resistant audit logging. Click any node to view implementation details.
Interactive Demo
Walk through the system design and data flow for this project step by step.
The state machine uses a Map state with MaxConcurrency=12 to execute all branches simultaneously. Each branch is an independent task with its own error handler. The execution ARN becomes the audit trail. When the SOC 2 auditor asks "prove request #4821 touched all twelve systems," the execution history answers that question directly.
Overview
Business Impact
What Went Wrong
KMS throttling nearly sank the encryption design. Calling
GenerateDataKey per field per record at 4M records/day exceeded the default request quota within minutes of the first production run. The fix was caching data keys at the Spark-partition level. This slightly enlarges the blast radius of a single data key and that trade-off is documented in the threat model.The MongoDB deletion branch taught me why idempotency was worth the effort. An early version deleted by a query that, on retry after a partial failure, matched documents belonging to a different customer with a colliding legacy identifier. It was caught in staging by a test I had almost not written.
Lessons Learned
I also underestimated the documentation half of SOC 2 by roughly a factor of two. The controls were the easy part; producing evidence that the controls had operated continuously across the observation window was its own workstream. Both low-severity observations in the final report were documentation gaps, not technical ones.
Technical Highlights
- Crypto-shredding via per-field KMS data keys: revoking a key renders that PII field unreadable across the entire data lake without rewriting files
- Consent management registry in PostgreSQL RDS tracking opt-in, opt-out, and withdrawal with timestamps and legal basis per data subject
- Right-to-erasure orchestration via Step Functions fanning out to 12 systems with retry, backoff, and partial-failure alerting
- Macie daily scans with custom data identifiers tuned to reduce UUID false positives from 30% to under 2%
- CloudTrail logs shipped to a separate audit AWS account that engineering cannot access, with integrity validation enabled
- Data residency enforcement via SCP policies blocking cross-region replication for EU citizen data in eu-west-1
- SOC 2 Type II controls mapped to CC6 (logical access), CC7 (system operations), CC8 (change management)
- Glue ETL jobs with PySpark doing schema validation, deduplication via deterministic hash, and field-level envelope encryption
- Data retention at the S3 lifecycle level: transition to Glacier after 90 days, deletion after 7 years
- Quarterly IAM access reviews automated via Lambda comparing role assignments against the HR system active employee list
- KMS data key caching per Glue partition to stay under the 10,000 requests/second API throttling limit
- MongoDB deletion branch required special idempotency handling because the analytics cluster had no unique delete endpoint
Want results like this for your infrastructure?
I specialize in taking complex AI pipelines and cloud setups from concept to high-availability production. Let's discuss how to optimize your workloads, secure your environment, and reduce cloud costs.