startpage results won't go past few pages #28

Closed
opened 2024-07-19 22:38:27 +00:00 by Evhorizon · 7 comments

Running the docker 4get container on localhost, when scraping with startpage the first one or two pages shows up but if I try to move on I get "The next page token is invalid or has expired!". Sometimes the "next page" bottom button doesn't show up at all and I get the same error if I refresh the search.

Running the docker 4get container on localhost, when scraping with startpage the first one or two pages shows up but if I try to move on I get "The next page token is invalid or has expired!". Sometimes the "next page" bottom button doesn't show up at all and I get the same error if I refresh the search.
Owner

Can you link me a search term that outputs this error?

Can you link me a search term that outputs this error?
Author
Yep http://localhost/?q=can%27t+open+display+%3A0&t=fpas&ia=web
Owner

I'm unable to reproduce on my own instance

https://4get.ca/web?s=can%27t+open+display+%3A0&scraper=startpage

The URL you linked me seems to come from somewhere else than 4get

I'm unable to reproduce on my own instance https://4get.ca/web?s=can%27t+open+display+%3A0&scraper=startpage The URL you linked me seems to come from somewhere else than 4get
Author

Huh sorry, my bad, I had launched the same search with DDG as well and I ended up mixing the URLs :P. The correct link is;
http://localhost/web?s=can%27t+open+display+%3A0
For what it's worth, I'm searching over Tor.

Huh sorry, my bad, I had launched the same search with DDG as well and I ended up mixing the URLs :P. The correct link is; http://localhost/web?s=can%27t+open+display+%3A0 For what it's worth, I'm searching over Tor.
Owner

I don't seem to understand your problem.

Startpage returns 4 pages of results on the official site, with the number of results per page set to 20 in the settings. 4get also uses this configuration.

I just tried to do a search with the tor proxy enabled, and sure enough, it does give me 4 pages of results over on 4get as well.

Getting results from google directly will give you more pages. This is just how startpage/google API works I guess.

For the invalid token error, you might've tried to refresh the page inadvertently while it was loading the next page. You can't request the same next page twice.

I don't seem to understand your problem. Startpage returns 4 pages of results on the official site, with the number of results per page set to 20 in the settings. 4get also uses this configuration. I just tried to do a search with the tor proxy enabled, and sure enough, it does give me 4 pages of results over on 4get as well. Getting results from google directly will give you more pages. This is just how startpage/google API works I guess. For the invalid token error, you might've tried to refresh the page inadvertently while it was loading the next page. You can't request the same next page twice.
Author

So something really weird is happening on my side. I get kind of 2 pages without issues, if I click on "next page" after that the result are loaded, I get the error. Another detail I forgot, when I open a link from the results and go back to the results page, the same error shows up again.

So something really weird is happening on my side. I get kind of 2 pages without issues, if I click on "next page" after that the result are loaded, I get the error. Another detail I forgot, when I open a link from the results and go back to the results page, the same error shows up again.
Author

Ok lets 4get this :) the issue has disappeared, maybe it was some kind of tor network temporary quirk.

Ok lets 4get this :) the issue has disappeared, maybe it was some kind of tor network temporary quirk.
Sign in to join this conversation.
No Label
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: lolcat/4get#28
No description provided.