Multiprocessing in a for loop for web-scraping

I am trying to do multiprocessing over the for loop. What I need help with is simultaneously looping different ranges. For example, one process must cover the range [1884463168 - 1884464000] and the other process simultaneously cover the range [1884464001 - 1884465000].

  • The code of the loop is incomplete, so that the post is not so long.
seriales = list(range(1884463168,1884464000,1))
lista = [str(x) for x in seriales]
base1 = []
variables = ['ID Base', 'ID Vacante']
for i in range(len(lista)):
     id_vacante = x
     id_base = y
     base1.append((id_vacante, id_base))
df = pd.DataFrame(base1, columns=variables)
df.to_csv('vacantes_2020_feb_mar.csv', index=False, encoding='utf-8')


Read more here: https://stackoverflow.com/questions/65056776/multiprocessing-in-a-for-loop-for-web-scraping

Content Attribution

This content was originally published by Juan Felipe Rubio Arrubla at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: