Porting over a BigQuery question from a customer:
Hi, I had quite a lot of trouble today with our pipelines, and one of this error was:
Job 30018: Subtask Lead_to_BigQuery Exception: BigQuery job failed. Final error was: {'reason': 'invalid', 'location': 'gs://prod_bucket_2/salesforce/20200512T000000/lead/records20.json', 'message': 'Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 26; errors: 1. Please look into the errors[] collection for more details.'}. ... {'reason': 'invalid', 'message': 'Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 26; errors: 1; max bad: 0; error percent: 0'}, {'reason': 'invalid', 'location': 'gs://prod_bucket_2/salesforce/20200512T000000/lead/records20.json', 'message': 'Error while reading data, error message: JSON parsing error in row starting at position 465142: Could not convert value to string. Field: company_revenue__c; Value: 7000000.0'}], 'state': 'DONE'}}
It seems like BigQuery couldn’t convert one of our Salesforce column into string. I know it’s not completely related to Airflow, but I’m quite lost so I was wondering if you had already encountered something like that and would be able to give me some pointers?
Thanks a lot in advance and stay safe!