raise JSONDecodeError(“Expecting value”, s, err.value)

I am trying to crawl data, however the code returns error with json.loads . When I traced back to the error, I realized that the element is None in loop so json.loads can not run. Is there any solution ?

Below is my code :

import json
from selenium import webdriver
import pandas as pd
from bs4 import BeautifulSoup
from datetime import datetime

start_time = datetime.now()


data = []
 
op = webdriver.ChromeOptions()
op.add_argument('--ignore-certificate-errors')
op.add_argument('--incognito')
op.add_argument('--headless')
driver = webdriver.Chrome(executable_path='D:/Desktop/Query/chromedriver.exe',options=op)
driver.get('https://www.cdiscount.com/f-1175520-MIS2008813786478.html')
link = 'https://www.cdiscount.com/f-1175520-MIS2008813786478.html'
soup = BeautifulSoup(driver.page_source, 'html.parser')
b = soup.prettify()
product_title = soup.find('title').getText()
reviews = soup.find_all("script",type="application/ld+json")
for element in reviews : 
     json_string = element.getText()
     json_dict = json.loads(json_string)
     data.append(json_dict)


Read more here: https://stackoverflow.com/questions/64948849/raise-jsondecodeerror-expecting-value-s-err-value

Content Attribution

This content was originally published by mht at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: