How to get more posts from Twitter gems?
I'm banging my head trying to figure out how Twitter pagination works.
I've tried max_id
and cursor
and they both don't work strangely.
Basically the maximum I can get from the search results is 100 and I would like to get 500.
Current code:
max_page = 5
max_id = -1
@data = []
for i in (1..max_page)
t = twt_client.search("hello world", :count => 100, :result_type => :recent, :max_id => max_id)
t.each do | tweet |
@data << tweet
end
max_id = t.next_results[:max_id]
end
This is actually telling me that next_results
is a private method, who has a working solution?
source to share
Not knowing what you are referring to pearls (indicate URL-address), I would say that cursor
, and max_id
will not get what you want. However count
. Since you say that you are only fetching 100 results, and count
- 100
, this makes sense to me.
t = twt_client.search("hello world", :count => 500, :result_type => :recent, :max_id => max_id)
I am assuming you are talking about the Twitter client referenced here . My first question is what twt_client
and in this regard, what search
does his method return ? It is also possible that you unwittingly updated the gem and there was a codebase change that made your current script out of date.
Take a look at your installed gem version and have a look at the README here .
source to share
Twitter :: SearchResults # next_results is private because they try to provide a consistent interface for enumeration.
Look, there was Twitter :: Enumerable included in search_results.rb
module Twitter
class SearchResults
include Twitter :: Enumerable
...
private
def last?
! next_page?
end
...
def fetch_next_page
response = @ client.send (@request_method, @path, next_page) .body
self.attrs = response
end
...
end
end
And if you look at enumerable.rb you will see this Twitter :: SearchResults # last? and Twitter :: SearchResults # fetch_next_page are used by Twitter :: SearchResults # each method
module Twitter
module Enumerable
include :: Enumerable
# @return [Enumerator]
def each (start = 0)
return to_enum (: each, start) unless block_given?
Array (@collection [start ..- 1]). Each do | element |
yield (element)
end
unless last?
start = [@ collection.size, start] .max
fetch_next_page
each (start, & Proc.new)
end
self
end
...
end
end
And Twitter :: SearchResults # will each iterate through the pages until Twitter responses have @attrs [: search_metadata] [: next_results]. So you need to break the iteration after reaching the 500th item.
I think you just need to use each
@data = []
tweet_number = 1
search_results = twt_client.search("hello world", count: 100, result_type: :recent)
search_results.each do |tweet|
@data << tweet
break if tweet_number == 500
end
This post is the result of exploring gem sources and the twitter api. I could be wrong somewhere since I didn't check my thoughts in the console.
source to share
Try this (I basically just updated the calculation max_id
in the loop):
max_page = 5
max_id = -1
@data = []
for i in (1..max_page)
t = twt_client.search("hello world", :count => 100, :result_type => :recent, :max_id => max_id)
t.each do | tweet |
@data << tweet
end
max_id = t.to_a.map(&:id).max + 1 # or may be max_id = t.map(&:id).max + 1
end
source to share