Dump tables without JDBC

[koliankolin]

Hello!

A bit of context. We plan to make data quality checks of CDC replication with Airbyte. For instance, we load huge table with CDC and once a day we load whole this table with Airbyte and count some data quality metrics. There're deletions in this table. Our table in MySQL and it is huge enough to load it with JDBC. Our connection is MySQL -> BigQuery.

Is there any other options besides JDBC to load large tables in Full refresh | Overwrite sync method? Or maybe there is any config options to use another tool? If I use GCS as intermediate storage will it help or it will also use JDBC but wrote JSON raw data into GCS and it will be particular the same?

[marcosmarxm]

Most databases connectors use JDBC driver to load data to destination.
Maybe one strategy you can do is to create views in your source which give you summary of the data you should retrieve using Airbyte and than it's possible to only get the summary and compare with Airbyte output.