lolcat
/
4get
6
13
Fork 4
Code Issues 13 Pull Requests Packages Projects Releases Wiki Activity

Request: add startpage scraper #23

New Issue
Closed
opened 2024-06-21 13:21:42 +00:00 by Evhorizon · 15 comments
Evhorizon commented 2024-06-21 13:21:42 +00:00
Copy Link

First of all thanks for all the great job you’re doing :) I really want to use the docker 4get+Tor container, but as I told elsewhere I’m mostly using the Google scraper, that is unforgiving blocking my searches with Tor. If startpage engine is added, it would fix my problem, because this engine proxies the Google results without blocking Tor. Thanks in advance, I hope this will be possible, tho I'm aware that each scraper is some thousands lines of code :)

First of all thanks for all the great job you’re doing :) I really want to use the docker 4get+Tor container, but as I told elsewhere I’m mostly using the Google scraper, that is unforgiving blocking my searches with Tor. If startpage engine is added, it would fix my problem, because this engine proxies the Google results without blocking Tor. Thanks in advance, I hope this will be possible, tho I'm aware that each scraper is some thousands lines of code :)
lolcat commented 2024-06-21 19:56:06 +00:00
Owner
Copy Link

I'm planning to add more Google sources soon, please stay tuned

I'm planning to add more Google sources soon, please stay tuned
👍 1
Evhorizon commented 2024-06-21 20:37:04 +00:00
Author
Copy Link

Thank you, I'll be on the lookout

Thank you, I'll be on the lookout
Evhorizon changed title from Request: add startpage to Request: add startpage scraper 2024-06-22 18:04:11 +00:00
Evhorizon commented 2024-06-25 16:21:54 +00:00
Author
Copy Link

In the meantime tried a workaround using the !sp bang in DDG but I was out of luck :) I get Failed to get d.js URL

In the meantime tried a workaround using the !sp bang in DDG but I was out of luck :) I get` Failed to get d.js URL`
lolcat commented 2024-06-25 19:03:12 +00:00
Owner
Copy Link

I'm working on a Startpage scraper, please stay tuned.

Just telling you now, their image search uses bing. I haven't checked for the video/news tab yet.

I'm working on a Startpage scraper, please stay tuned. Just telling you now, their image search uses bing. I haven't checked for the video/news tab yet.
👍 1
Evhorizon commented 2024-06-25 19:31:58 +00:00
Author
Copy Link

Thanks:) huh, Bing, really? Damn, so which engine is left sourcing Google results? Gibiru?

Thanks:) huh, Bing, really? Damn, so which engine is left sourcing Google results? Gibiru?
lolcat commented 2024-06-25 20:01:08 +00:00
Owner
Copy Link

I don't know why people keep mentioning Gibiru as an option. It's literally just reskinned Google CSE. Open the network tab, you will see it makes calls to cse.google.com to pull results, it's literal garbage.

There are a few Google options out there, like MyPrivateSearch which has a hidden Google API endpoint and some other ones with shitty reputation. Once I scrape Startpage, I'm thinking about writing scrapers for Korean and Chinese search engines, they have interesting results.

I don't know why people keep mentioning Gibiru as an option. It's literally just reskinned Google CSE. Open the network tab, you will see it makes calls to cse.google.com to pull results, it's literal garbage. There are a few Google options out there, like MyPrivateSearch which has a hidden Google API endpoint and some other ones with shitty reputation. Once I scrape Startpage, I'm thinking about writing scrapers for Korean and Chinese search engines, they have interesting results.
👍 1
Evhorizon commented 2024-06-25 20:06:43 +00:00
Author
Copy Link

Yes, Gibiru is known to be a bare Google clone, that's why I mentioned it hoping it would not have blocked me:) however Asian engines sound interesting, are you talking about something like Baidu?

Yes, Gibiru is known to be a bare Google clone, that's why I mentioned it hoping it would not have blocked me:) however Asian engines sound interesting, are you talking about something like Baidu?
lolcat commented 2024-06-25 21:05:23 +00:00
Owner
Copy Link

I have a few in my list:

Baidu, Seznam, Daum, Sese, m.sm.cn, Toutiao, Naver, Sogou, So.com, Coc Coc, Goo, Weibo, Solofield.net

Just to name a few.

I have a few in my list: Baidu, Seznam, Daum, Sese, m.sm.cn, Toutiao, Naver, Sogou, So.com, Coc Coc, Goo, Weibo, Solofield.net Just to name a few.
Evhorizon commented 2024-06-25 21:09:51 +00:00
Author
Copy Link

They will all come as a surprise, honestly I haven't ever heard of any of those :)

They will all come as a surprise, honestly I haven't ever heard of any of those :)
lolcat commented 2024-06-25 21:14:17 +00:00
Owner
Copy Link

I've been hunting for them autistically

I've been hunting for them autistically
👀 1
lolcat commented 2024-07-15 01:49:57 +00:00
Owner
Copy Link

I've been doing some work on it lately, expect an update on this sooner or later

I've been doing some work on it lately, expect an update on this sooner or later
❤️ 1
lolcat commented 2024-07-15 03:17:26 +00:00
Owner
Copy Link

I have added partial support for startpage with my latest commit.

Image, videos and news will follow soon...

still ironing out some issues with the instant answers that suddenly stopped showing up. what a pain

edit: ^ just fixed whatever bs that was

I have added partial support for startpage with my latest commit. Image, videos and news will follow soon... still ironing out some issues with the instant answers that suddenly stopped showing up. what a pain edit: ^ just fixed whatever bs that was
❤️ 1
Evhorizon commented 2024-07-15 06:58:09 +00:00
Author
Copy Link

Great job man, felt like something exciting was going to happen this weekend:) I'm waiting for the docker code to be updated and try it asap

Great job man, felt like something exciting was going to happen this weekend:) I'm waiting for the docker code to be updated and try it asap
Evhorizon commented 2024-07-15 07:02:44 +00:00
Author
Copy Link

Image, videos and news will follow soon...

Good, my personal priority would go to image and news because yt is already a satisfying choice to look for videos

> Image, videos and news will follow soon... Good, my personal priority would go to image and news because yt is already a satisfying choice to look for videos
lolcat commented 2024-07-29 14:27:15 +00:00
Owner
Copy Link

I fully implemented startpage in my latest commit, thank you for your time

I fully implemented startpage in my latest commit, thank you for your time
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: lolcat/4get#23
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?