How do I scrape multi-language web using Python

I'm using Python to scrape data from Japanese website where it offers both English & Japanese language. Link here

The problem is I got the data I needed but in the wrong language (Links of both languages are identical). I tried inspecting the html page and saw the element 'lang' as followed:

<html xmlns="" lang="en" xml:lang="en">

<html xmlns="" lang="ja" xml:lang="ja" class="">

Here is the code I used:

import requests
import lxml.html as lh
import pandas as pd

page = requests.get(url)
doc = lh.fromstring(page.content)
tr_elements = doc.xpath('//tr')
col = []
i = 0

for t in tr_elements[0]:
    i += 1
    name = t.text_content()

At this point I got the head row of the table from the page but in Japanese version. I'm new to Python and the scrapy. I don't know if there's any method I could use to get the data in English? If there is any existing examples, templates or other resources I can could use, that'd be better.

Thanks in advance!

Read more here:

Content Attribution

This content was originally published by user1829525 at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: