Hi ,
I need some suggestions related to setting up scrapers with Airflow.
Issue is lets say i have a website to scrape that has some 3k links. I want to divide this over 3 batches of 1k each and all these have to run on different days .
What can be the best approach in this case? If we can do some conditional parameter basis scheduling. Example I have excel as my data source so next to every url i can mention like Batch no … now basis batch number if we can schedule differently?
Hope this makes some sense. Please suggest