Write multiple tables to bigquery by filtering

I am reading from a jdbc connection converting to tablerows and then uploading the data to bigquery with the following code.

The data I am expecting has multiple columns, one of them called "Phone number". If I want to store rows where this phone number is less than 8 digits long in a different table, remove them, and write both tables (one with invalid phone numbers and the other with the remaining data with correct phone numbers) to bigquery. What method can I use after reading the jdbc data to do this?

  private static PipelineResult run(JdbcConverters.JdbcToBigQueryOptions options) {

    // Create the pipeline
    Pipeline pipeline = Pipeline.create(options);

     * Steps: 1) Read records via JDBC and convert to TableRow via RowMapper
     *        2) Filter data
     *        3) Append TableRow to BigQuery via BigQueryIO
         * Step 1: Read records via JDBC and convert to TableRow
         *         via {@link org.apache.beam.sdk.io.jdbc.JdbcIO.RowMapper}
            "Read from JdbcIO",
                            maybeDecrypt(options.getConnectionURL(), options.getKMSEncryptionKey()))
                            maybeDecrypt(options.getUsername(), options.getKMSEncryptionKey()))
                            maybeDecrypt(options.getPassword(), options.getKMSEncryptionKey()))

            "Write to BigQuery",

    // Execute the pipeline and return the result.
    return pipeline.run();

Read more here: https://stackoverflow.com/questions/68491825/write-multiple-tables-to-bigquery-by-filtering

Content Attribution

This content was originally published by john smith at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: