https://ccharlesgb.github.io/cc-portfolio/tags/bigquery/Recent content in Bigquery on Connor CharlesHugoen-usTue, 14 Jul 2026 00:00:00 +0000https://ccharlesgb.github.io/cc-portfolio/post/transferring-billion-row-datasets-from-bigquery-into-postgres/Tue, 14 Jul 2026 00:00:00 +0000https://ccharlesgb.github.io/cc-portfolio/post/transferring-billion-row-datasets-from-bigquery-into-postgres/<p>A common challenge in data engineering is figuring out how to move a large dataset in one structure, format or cloud region and transform it into another without creating a large cloud bill or having to wait a long time for a batch job to complete. At Autotrader, we do a lot of batch processing using <a href="https://www.getdbt.com/">dbt</a> and BigQuery. This often results in datasets that we want to then export to operational datastores such as Postgres or Mongo. We then surface these datasets in embedded analytics products, which require the low latency that Postgres and Mongo can deliver.</p>