Google served a GPDR consent form #5
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
I started receiving this error on one of my proxies.
This scraper returned an error:
Google served a GPDR consent form. This should not happen, please report if you encounter this message
My 4get instance is password protected and I'm the only one using it. It hasn't been used much, but it was working earlier. The only thing I can think that might be odd is I reinstalled that server a few times and the public IPv6 changed a few times, but still in the same /64.
I was rate limited on my main instance without a proxy earlier, too, with only me searching. It cleared the next day, though. We'll see if it's the same with this.
Hey!
Can you make sure you're using the latest commit? I think I fixed this error message, now it should throw a "4get blocked this instance" error.
... I will need to rewrite the google scraper anyway (rip... 4k lines of code!) because the method I've been using is starting to fail. They're doing A/B testing to switch to a new user interface... Which breaks everything. I must write a scraper to parse the new old-mobile-device interface.
I just did a pull; it's already up to date. I just installed it last week. In any case, whatever the error, the result is that it's blocked. Just wondering if it's something especially different or scary about being a GPDR error.
If they're changing the search interface, that might explain why the google searches in general fail maybe 10% of the time, but clicking search again works.
That's... Really weird. To bypass the consent form, I'm passing the following cookie to tell Google we've already accepted the terms:
SOCS=CAESNQgCEitib3FfaWRlbnRpdHlmcm9udGVuZHVpc2VydmVyXzIwMjQwMzE3LjA4X3AwGgJlbiAEGgYIgM7orwYWhen decoded to base64, it gave some message but I couldn't be bothered to decode that crap, so I just hardcoded it in. Currently using european proxies on 4get.ca with no issues using the token from above, so I don't know why you're having issues here...
On your server, would it be possible to setup a reverse proxy, set your user agent to
Mozilla/5.0 (Linux; U; Android 2.3.3; pt-pt; LG-P500h-parrot Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 MMS/LG-Android-MMS-V1.0/1.2and search something up on Google? Do you still get the GPDR redirect then?If so, can you please accept the terms and give me your SOCS cookie so I can investigate further?
I did connect to the proxy that's giving the error from Chromium with the user agent changed, but I don't get any error searching google directly like that.
The server is in Singapore, for what it's worth.
Update:
Sorry, the user agent change wasn't sticking before. I tested again, and going to google.com redirects to:
https://www.google.cn/m
And gives a 404 error:
If I go to google.com/m instead of google.com it redirects me to:
https://www.google.com.hk/m
And from there, I'm able to search successfully without getting the error.
I did some more testing and changed the proxy configuration from socks5_hostname to socks5, so the dns resolution is done locally. And that is working without error now.
Maybe it's some sort of messed up google geolocation on the Singapore proxy. Anyway, local dns resolution seems to work around the issue for now.
Once I can get my hands on a Singapore proxy, I will be fixing this issue
keeping this issue open in the meantime, thank you for your time
Hey, I did a big update on the Google scraper, please tell me if the issue persists.