Scrape directory from web server

On websites without "index.html" files it is possible to see the other files in the server-side directory the url directs to from a browser. Is it possible to examine this information from a script, for a directory which does contain an "index.html" file without having downloaded the webpage?

I assume it's possible in selenium, but would like to avoid it if possible due to speed.

If so, how? If not, why - what goes on when you download a webpage from a browser and get the content?

Cheers!



Read more here: https://stackoverflow.com/questions/66343453/scrape-directory-from-web-server

Content Attribution

This content was originally published by kendfss at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: