Dealing with HTTP IncompleteRead Error in Python
I am trying to figure out how to handle http.client.IncompleteRead Error
in the code below. I am handling the error using the idea in this post . Basically I thought it was just a server limiting the number of times I can access the data, but oddly enough, sometimes I get an HTTP status code of 200, but still the code below returns a type None
. Is it because zipdata = e.partial
it returns nothing when an error occurs?
def update(self, ABBRV):
if self.ABBRV == 'cd':
try:
my_url = 'http://www.bankofcanada.ca/stats/results/csv'
data = urllib.parse.urlencode({"lookupPage": "lookup_yield_curve.php",
"startRange": "1986-01-01",
"searchRange": "all"})
binary_data = data.encode('utf-8')
req = urllib.request.Request(my_url, binary_data)
result = urllib.request.urlopen(req)
print('status:: {},{}'.format(result.status, my_url))
zipdata = result.read()
zipfile = ZipFile(BytesIO(zipdata))
df = pd.read_csv(zipfile.open(zipfile.namelist()[0]))
df = pd.melt(df, id_vars=['Date'])
return df
#In case of http.client.IncompleteRead Error
except http.client.IncompleteRead as e:
zipdata = e.partial
thank
source to share
Hmm ... I ran your code 20 times without getting an incomplete read error, why don't you just try again in case of an incomplete read? Or on the other hand, if your IP gets blocked, then it would make sense that they don't return anything to you. Your code might look something like this:
maxretries = 3
attempt = 0
while attempt < maxretries:
try:
#http access code goes in here
except http.client.IncompleteRead:
attempt += 1
else:
break
source to share