forked from lolcat/4get
1
0
Fork 0

Compare commits

..

56 Commits

Author SHA1 Message Date
lolcat 78aa2e198f fuiwhwehfuiewuf 2025-04-19 10:42:21 -04:00
lolcat b85820cbcd hopefully this fixes bing images my fucking god 2025-04-19 10:37:35 -04:00
lolcat 4b85841a3e forgot nsfw filter god damn 2025-04-17 21:41:12 -04:00
lolcat 8d07e72dfe forgot settings, god damn i have dementia 2025-04-17 20:08:15 -04:00
lolcat 566680fe36 ok its unfucked now 2025-04-17 20:06:42 -04:00
lolcat 077692db49 i fucking hate bing 2025-04-17 20:05:58 -04:00
lolcat e4bf53cdaa forgot config 2025-04-17 20:02:02 -04:00
lolcat 4489bb21e5 forgot flickr 2025-04-17 20:00:33 -04:00
lolcat ff8b1addf7 fixed bing images failing to load, added flickr 2025-04-17 19:54:34 -04:00
lolcat 3e2c3fc5d9 fixed google videos 2025-04-02 21:40:53 -04:00
lolcat 49ddd1a216 duckduckgo images nsfw fix 2025-03-20 21:05:36 -04:00
lolcat 81ca8eaddc gore theme fix 2025-03-16 03:39:58 -04:00
lolcat c9c8d578f3 Merge branch 'master' of https://git.lolcat.ca/lolcat/4get 2025-03-02 21:58:34 -05:00
lolcat b2203804c7 path traversal exploit (this is what you get for using free software) 2025-03-02 21:58:18 -05:00
lolcat 13dfa9240c Merge pull request 'fix Dockerfile build.' (#67) from Fijxu/4get:dockerfile-fix into master
Reviewed-on: lolcat/4get#67
2025-02-04 07:05:12 +00:00
Fijxu 0a53c3605a
fix Dockerfile build.
The `alpine:latest` image do not longer include php83 on their repos.
Using a specific image tag is better to prevent breakages on the future.

Ref: https://github.com/dnaprawa/dockerfile-best-practices?tab=readme-ov-file#the-latest-is-an-evil-choose-specific-image-tag
2025-02-02 01:37:14 -03:00
lolcat 36b0c570aa Merge branch 'master' of https://git.lolcat.ca/lolcat/4get 2025-02-01 22:53:06 -05:00
lolcat 47a7a2a224 brave pagination fix 2025-02-01 22:51:44 -05:00
lolcat 0180cf5224 Merge pull request 'override ssl.conf when using http config' (#63) from docker_fix_conf into master
Reviewed-on: lolcat/4get#63
2025-01-24 15:41:41 +00:00
throwaway eed32a153c remove ssl.conf when using http config 2025-01-24 00:53:34 -08:00
lolcat f9f3c919d6 brave scraper fix 2025-01-22 20:04:42 -05:00
lolcat 4b0d8f75dc google scraper fix 2025-01-19 14:02:24 -05:00
lolcat 033e4cb959 added vsco scraper 2025-01-11 23:07:58 -05:00
lolcat 91f621e105 readme changes 2025-01-11 14:34:23 -05:00
lolcat 9f60900875 500px scraper 2025-01-11 14:12:54 -05:00
lolcat 631aa58565 marginalia hotfix 2025-01-07 21:12:07 -05:00
lolcat b892f90b13 readme changes 2025-01-07 00:05:42 -05:00
lolcat 463ba0775f added pinterest 2025-01-06 23:56:49 -05:00
lolcat cfad4fb035 wordnik bugfix 2025-01-06 21:05:45 -05:00
lolcat 4e968b4b1c youtube scraper fix 2025-01-03 22:22:30 -05:00
lolcat 81df52235c fixed google crash 2025-01-03 21:43:40 -05:00
lolcat 1ca2626ad9 fix ddg bug with EOF result 2025-01-03 21:16:00 -05:00
lolcat 9ca93f34c6 ddg hotfix 2024-12-17 21:01:36 -05:00
lolcat 0a43b9c849 added arquivo.pt 2024-12-17 10:11:53 -05:00
lolcat b636fec319 fucking git is so shit 2024-12-17 00:35:15 -05:00
lolcat 774f7113df duckduckgo scraper rewrite 2024-12-17 00:31:15 -05:00
lolcat 0b3bbe0f15 gore's shitty theme fix 2024-12-02 15:19:10 -05:00
lolcat 5f0b0a7b83 findthatmeme fix 2024-12-01 15:59:03 -05:00
lolcat 920b9d5b3f brave crash fix 2024-11-19 09:22:58 -05:00
lolcat 9cd369ac08 http2 on ddg 2024-11-07 23:37:43 -05:00
lolcat e83865be49 added pagination 2024-11-07 00:12:06 -05:00
lolcat 68dd7f29f6 mojeek thumbnail fix 2024-11-06 23:43:54 -05:00
lolcat aaa30c79f5 fix google cse image crash + added word autocorrect 2024-10-31 20:31:23 -04:00
lolcat 070f9d442b brave fix 2024-10-29 21:29:17 -04:00
lolcat 9c18753ec3 yes of course i need to fucking forget the .php again AAAAAAAAAAA 2024-10-24 21:46:54 -04:00
lolcat d8a729796e fix crash on google cse, added settings 2024-10-22 20:15:00 -04:00
lolcat 2bbe5a29a9 Merge branch 'master' of https://git.lolcat.ca/lolcat/4get 2024-10-22 11:34:06 -04:00
lolcat 9ac195ac3b added google CSE 2024-10-22 11:33:14 -04:00
lolcat d427a48ed4 Merge pull request 'nginx documentation but better' (#41) from bread/4get:master into master
Reviewed-on: lolcat/4get#41
2024-10-21 14:17:07 +00:00
lolcat 12d5b4ade8 Merge branch 'master' into master 2024-10-21 14:16:54 +00:00
Pano c422abbdc6 add css via copy and paste (slightly edited to not require a lot from my shithole) 2024-10-21 14:15:34 +00:00
Pano 85246cc7ec 194 lines of mark(down) 2024-10-19 01:12:41 +00:00
Pano d709d12111 fix html making md look bad 2024-10-10 04:45:06 +00:00
Pano 19f82a8536 even more little things I missed (polish!) 2024-10-08 19:22:38 +00:00
Pano 155a38d454 things I missed 2024-10-08 19:19:01 +00:00
Pano 6926e374af Upgrade nginx configuration to a better state 2024-10-07 23:48:24 +00:00
34 changed files with 5776 additions and 5987 deletions

1
.dockerignore Normal file
View File

@ -0,0 +1 @@
.git

View File

@ -1,10 +1,9 @@
FROM alpine:latest
FROM alpine:3.21
WORKDIR /var/www/html/4get
RUN apk update && apk upgrade
RUN apk add php apache2-ssl php83-fileinfo php83-openssl php83-iconv php83-common php83-dom php83-sodium php83-curl curl php83-pecl-apcu php83-apache2 imagemagick php83-pecl-imagick php-mbstring imagemagick-webp imagemagick-jpeg
RUN apk add php apache2-ssl php84-fileinfo php84-openssl php84-iconv php84-common php84-dom php84-sodium php84-curl curl php84-pecl-apcu php84-apache2 imagemagick php84-pecl-imagick php84-mbstring imagemagick-webp imagemagick-jpeg
COPY ./docker/apache/ /etc/apache2/
COPY . .
RUN chmod 777 /var/www/html/4get/icons
@ -14,4 +13,5 @@ EXPOSE 443
ENV FOURGET_PROTO=http
CMD ["./docker/docker-entrypoint.sh"]
ENTRYPOINT ["./docker/docker-entrypoint.sh"]
CMD ["start"]

View File

@ -9,9 +9,11 @@ https://4get.ca/about
## Official instance
https://4get.ca , or visit the official instance list: https://4get.ca/instances
_NOT to be confused with 4get.ch, 4get.lol and friends! I **don't** host these._
## Totally unbiased comparison between alternatives
| | 4get | searx(ng) | libreY | araa | hearch |
| | 4get | searx(ng) | libreY | araa | hearch.co |
|----------------------------|-------------------------|-----------|-------------|-----------|-------------------|
| RAM usage | 200-400mb~ | 2GB~ | 200-400mb~ | 2GB~ | idk |
| Does it suck | no (debunked by snopes) | yes | yes | a little | better than searx |
@ -23,9 +25,9 @@ https://4get.ca , or visit the official instance list: https://4get.ca/instances
3. Bot protection that *actually* filters out the bots (when configured)
4. Interface doesn't require javascript
5. Favicon fetcher with caching support & image proxy
6. Bunch of other shit
6. Bunch of other shits
tl;dr the best way to actually browse for shit.
tl;dr 4get is the best way to browse for shit.
# Supported websites
@ -39,11 +41,11 @@ tl;dr the best way to actually browse for shit.
| Qwant | Qwant | Startpage | Mojeek | | Kagi |
| Ghostery | Yep | Qwant | | | Qwant |
| Yep | Solofield | Solofield | | | Ghostery |
| Greppr | Imgur | | | | Yep |
| Crowdview | FindThatMeme | | | | Marginalia |
| Mwmbl | | | | | YouTube |
| Mojeek | | | | | Soundcloud |
| Solofield | | | | | |
| Greppr | Pinterest | | | | Yep |
| Crowdview | 500px | | | | Marginalia |
| Mwmbl | VSCO | | | | YouTube |
| Mojeek | Imgur | | | | Soundcloud |
| Solofield | FindThatMeme | | | | |
| Marginalia | | | | | |
| wiby | | | | | |
| Curlie | | | | | |
@ -52,7 +54,7 @@ tl;dr the best way to actually browse for shit.
Refer to the <a href="https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/">documentation index</a>. I recommend following the <a href="https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/apache2.md">apache2 guide</a>.
## Contact
Shit breaks all the time but I repair it all the time too... Email me here: <b>will (at) lolcat.ca</b> or create an issue.
Shit breaks all the time but I repair it all the time too. Email me here: <b>will (at) lolcat.ca</b> or create an issue.
## License
AGPL

19
api.txt
View File

@ -1,9 +1,16 @@
__ __ __
/ // / ____ ____ / /_
/ // /_/ __ `/ _ \/ __/
/__ __/ /_/ / __/ /_
/_/ \__, /\___/\__/
/____/
44
4444444 44
44444444 44444 444
44444444 444444 444444444
44444 44444444 444444444
444444444 4444444
4444444444 444444
4444444444444
444444444444444444
444444444444444
44444444
4444
44
+ Welcome to the 4get API documentation +

View File

@ -119,7 +119,7 @@ class config{
// Default user agent to use for scraper requests. Sometimes ignored to get specific webpages
// Changing this might break things.
const USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:129.0) Gecko/20100101 Firefox/129.0";
const USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0";
// Proxy pool assignments for each scraper
// false = Use server's raw IP
@ -129,6 +129,8 @@ class config{
const PROXY_BRAVE = false;
const PROXY_FB = false; // facebook
const PROXY_GOOGLE = false;
const PROXY_GOOGLE_API = false;
const PROXY_GOOGLE_CSE = false;
const PROXY_STARTPAGE = false;
const PROXY_QWANT = false;
const PROXY_GHOSTERY = false;
@ -142,6 +144,10 @@ class config{
const PROXY_YT = false; // youtube
const PROXY_YEP = false;
const PROXY_PINTEREST = false;
const PROXY_SANKAKUCOMPLEX = false;
const PROXY_FLICKR = false;
const PROXY_FIVEHPX = false;
const PROXY_VSCO = false;
const PROXY_SEZNAM = false;
const PROXY_NAVER = false;
const PROXY_GREPPR = false;
@ -157,6 +163,9 @@ class config{
// Scraper-specific parameters
//
// GOOGLE CSE & GOOGLE API
const GOOGLE_CX_ENDPOINT = "d4e68b99b876541f0";
// MARGINALIA
// Use "null" to default out to HTML scraping OR specify a string to
// use the API (Eg: "public"). API has less filters.

View File

@ -6,14 +6,15 @@ services:
image: luuul/4get:latest
restart: unless-stopped
environment:
- FOURGET_PROTO=http
- FOURGET_SERVER_NAME=4get.ca
- FOURGET_INSTANCES=https://4get.ca
ports:
- "80:80"
- "443:443"
volumes:
- /etc/letsencrypt/live/domain.tld:/etc/4get/certs
# mount custom banners and captcha
- ./banners:/var/www/html/4get/banner
- ./captcha:/var/www/html/4get/data/captcha
# volumes:
# - /etc/letsencrypt/live/domain.tld:/etc/4get/certs # mount ssl
# - ./banners:/var/www/html/4get/banner # mount custom banners
# - ./captcha:/var/www/html/4get/data/captcha # mount captcha images

View File

@ -0,0 +1 @@
# intentionally blank

View File

@ -8,18 +8,27 @@ FOURGET_PROTO="${FOURGET_PROTO#\"}"
# make lowercase
FOURGET_PROTO=`echo $FOURGET_PROTO | awk '{print tolower($0)}'`
FOURGET_SRC='/var/www/html/4get'
mkdir -p /etc/apache2
if [ "$FOURGET_PROTO" = "https" ]; then
echo "Using https configuration"
cp /etc/apache2/https.conf /etc/apache2/httpd.conf
cp -r ${FOURGET_SRC}/docker/apache/https/httpd.conf /etc/apache2
cp -r ${FOURGET_SRC}/docker/apache/https/conf.d/* /etc/apache2/conf.d
else
echo "Using http configuration"
cp /etc/apache2/http.conf /etc/apache2/httpd.conf
cp -r ${FOURGET_SRC}/docker/apache/http/httpd.conf /etc/apache2
cp -r ${FOURGET_SRC}/docker/apache/http/conf.d/* /etc/apache2/conf.d
fi
php ./docker/gen_config.php
echo "4get is running"
exec httpd -DFOREGROUND
if [ "$@" = "start" ]; then
echo "4get is running"
exec httpd -DFOREGROUND
else
exec "$@"
fi

View File

@ -1,27 +1,67 @@
# Install on NGINX
<h1 align=center>Installation of 4get in NGINX</h1>
>I do NOT recommend following this guide, only follow this if you *really* need to use nginx. I recommend you use the apache2 steps instead.
<div align=right>
Login as root.
> NOTE: As the previous version stated, it is better to follow the <a href="https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/apache2.md">Apache2 guide</a> instead of the Nginx one.
Create a file in `/etc/nginx/sites-avaliable/` called `4get.conf` or any name you want and put this into the file:
> NOTE: This is going to guess that you're using either a <abbr title="(Arch Linux, Artix Linux, Endeavouros, etc...) ">Arch-based system</abbr> or a <abbr title="(Debian, Ubuntu, Devuan, etc...)">Debian-based system</abbr>, although you can still follow it with minor issues.
```
server {
# DO YOU REALLY NEED TO LOG SEARCHES?
access_log /dev/null;
error_log /dev/null;
# Change this if you have 4get in other folder.
</div>
1. Login as root.
2. Upgrade your system:
* On Arch-based, run `pacman -Syu`.
* On Debian-based, run `apt update`, then `apt upgrade`.
3. Install the following dependencies:
* `git`: So you can clone <a href="https://git.lolcat.ca/lolcat/4get">this</a> repository.
* `nginx`: So you can run Nginx.
* `php-fpm`: This is what allows Nginx to run *(and show)* PHP files.
* `php-imagick`, `imagemagick`: Image manipulation.
* `php-apcu`: Caching module.
* `php-curl`, `curl`: Transferring data with URLs.
* `php-mbstring`: String utils.
* `certbot`, `certbot-nginx`: ACME client. Used to create SSL certificates.
* In Arch-based distributions:
* `pacman -S nginx certbot php-imagick certbot-nginx imagemagick curl php-apcu git`
* In Debian-based distributions:
* `apt install php-mbstring nginx certbot-nginx certbot php-imagick imagemagick php-curl curl php-apcu git`
<div align=right>
> IMPORTANT: `php-curl`, `php-mbstring` might be a Debian-only package, but this needs further fact checking.
> IMPORTANT: If having issues with `php-apcu` or `libsodium`, go to [^1].
</div>
4. `cd` to `/etc/nginx` and make the `conf.d/` directory if it doesn't exist:
* Again, this guesses you're logged in as root.
```sh
cd /etc/nginx
ls -l conf.d/ # If ls shows conf.d, then it means it exists.
# If it does not, run:
mkdir conf.d
```
5. Make a file inside `conf.d/` called `4get.conf` and place the following content:
* First run `touch conf.d/4get.conf` then `nano conf.d/4get.conf` to open the nano editor: *(Install it if it is not, or use another editor.)*
```sh
server {
access_log /dev/null; # Search log file. Do you really need to?
error_log /dev/null; # Error log file.
# Change this if you have 4get in another folder.
root /var/www/4get;
# Change yourdomain by your domain lol
# Change 'yourdomain' to your domain.
server_name www.yourdomain.com yourdomain.com;
# Port to listen to.
listen 80;
location @php {
try_files $uri.php $uri/index.php =404;
# Change the unix socket address if it's different for you.
fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
fastcgi_index index.php;
# Change this to `fastcgi_params` if you use a debian based distro.
# Change this to `fastcgi_params` if you use a debian based distribution.
include fastcgi.conf;
fastcgi_intercept_errors on;
}
@ -34,56 +74,96 @@ server {
return 301 $1;
}
listen 80;
}
```
}
```
* The above is a very basic configuration and thus will need tweaking to your personal needs. It should still work as-is, though. A 'real world' example is present in [^2].
* After saving the file, check that the `nginx.conf` file inside the main directory includes files inside `conf.d/`:
* It should be inside the the http block: *(The following is an example! Don't just Copy and Paste it!)*
```sh
http {
include mime.types;
include conf.d/*.conf;
types_hash_max_size 4096;
# ...
}
```
* Now, test your configuration with `nginx -t`, if it says that everything is good, restart *(or start)* the Nginx daemon:
* This depends on the init manager, most distributions use `systemd`, but it's better practice to include most.
```sh
# systemd
systemctl stop nginx
systemctl start nginxt
# or
systemctl restart nginx
That is a very basic config so you will need to adapt it to your needs in case you have a more complicated nginx configuration. Anyways, you can see a real world example [here](https://git.zzls.xyz/Fijxu/etc-configs/src/branch/selfhost/nginx/sites-available/4get.zzls.xyz.conf)
# openrc
rc-service nginx stop
rc-service nginx start
# or
rc-service nginx restart
After you save the file you will need to do a symlink of the `4get.conf` file to `/etc/nignx/sites-enabled/`, you can do it with this command:
# runit
sv down nginx
sv up nginx
# or
sv restart nginx
```sh
ln -s /etc/nginx/sites-available/4get.conf /etc/nginx/sites-available/4get.conf
```
# s6
s6-rc -d change nginx
s6-rc -u change nginx
# or
s6-svc -r /run/service/nginx
Now test the nginx config with `nginx -t`, if it says that everything is good, restart nginx using `systemctl restart nginx`
# dinit
dinitctl stop nginx
dinitctl start nginx
# or
dinitctl restart nginx
```
6. Clone the repository to `/var/www`:
* `git clone --depth 1 https://git.lolcat.ca/lolcat/4get 4get` - It clones the repository with the depth of one commit *(so it takes less time to download)* and saves the cloned repository as '4get'.
7. That should be it! There are some extra steps you can take, but it really just depends on you.
# Encryption setup
<h2 align=center>Encryption setup</h2>
Generate a certificate for the domain using:
1. Generate a certificate for the domain you're using with:
* Note that `certbot-nginx` is needed.
```sh
certbot --nginx --key-type ecdsa -d www.yourdomain.com -d yourdomain.com
```
2. After that, certbot will deploy the certificate automatically to your 4get conf file; It should be ready to use from there.
```sh
certbot --nginx --key-type ecdsa -d www.yourdomain.com -d yourdomain.com
```
(Remember to install the nginx certbot plugin!!!)
<h2 align=center>Tor Setup</h2>
After doing that certbot should deploy the certificate automatically into your 4get nginx config file. It should be ready to use at that point.
<div align=right>
# Tor setup on NGINX
> IMPORTANT: Tor onion addresses are very long compared to traditional domains, so, Before doing anything, edit `nginx.conf` and increase <abbr title="This setting in your Nginx configuration controls the internal data structure used to manage multiple server names (hostnames) associated with your web server. Each hostname requires a certain amount of memory within this structure. If the size is insufficient, Nginx will encounter errors."><code>server_names_hash_bucket_size</code></abbr> to your needs.
Important Note: Tor onion addresses are significantly longer than traditional domain names. Before proceeding with Nginx configuration, ensure you increase the `server_names_hash_bucket_size` value in your `nginx.conf` file. This setting in your Nginx configuration controls the internal data structure used to manage multiple server names (hostnames) associated with your web server. Each hostname requires a certain amount of memory within this structure. If the size is insufficient, Nginx will encounter errors.
</div>
1. Open your `nginx.conf` file (that is under `/etc/nginx/nginx.conf`).
2. Find the line containing `# server_names_hash_bucket_size 64;`.
3. Uncomment the line and adjust the value. Start with 64, but if you encounter issues, incrementally increase it (e.g., 128, 256) until it accommodates your configuration.
1. `cd` to `/etc/nginx` *(if you haven't)* and open your `nginx.conf` file.
2. Find the line containing `# server_names_hash_bucket_size 64;` inside said file.
3. Uncomment the line and adjust the value; start with 64, but if you encounter issues, incrementally increase it *(e.g., 128, 256)* until it accommodates your configuration.
4. Open *(or duplicate the configuration)* and edit it:
* Example configuration, again:
```sh
server {
access_log /dev/null; # Search log file. Do you really need to?
error_log /dev/null; # Error log file.
Open your current 4get NGINX config (that is under `/etc/nginx/sites-available/`) and append this to the end of the file:
```
server {
access_log /dev/null;
error_log /dev/null;
listen 80;
server_name <youronionaddress>;
# Change this if you have 4get in another folder.
root /var/www/4get;
# Change 'onionadress.onion' to your onion link.
server_name onionadress.onion;
# Port to listen to.
listen 80;
location @php {
try_files $uri.php $uri/index.php =404;
# Change the unix socket address if it's different for you.
fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
fastcgi_index index.php;
# Change this to `fastcgi_params` if you use a debian based distro.
# Change this to `fastcgi_params` if you use a debian based distribution.
include fastcgi.conf;
fastcgi_intercept_errors on;
}
@ -95,9 +175,20 @@ server {
location ~* ^(.*)\.php$ {
return 301 $1;
}
}
```
Obviously replace `<youronionaddress>` by the onion address of `/var/lib/tor/4get/hostname` and then check if the nginx config is valid with `nginx -t` if yes, then restart the nginx service and try opening the onion address into the Tor Browser. You can see a real world example [here](https://git.zzls.xyz/Fijxu/etc-configs/src/branch/selfhost/nginx/sites-available/4get.zzls.xyz.conf)
}
```
A real world example is present in [^2].
5. Once done, check the configuration with `nginx -t`. If everything's fine and dandy, refer to <a href="https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/tor.md">the Tor guide</a> to setup your onion site.
Once you did the above, refer to <a href="https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/tor.md">this tor guide</a> to setup your onionsite.
<h2 align=center>Other important things</h2>
1. <a href="https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/configure.md">Configuration guide</a>: Things to do after setup.
2. <a href="https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/apache2.md">Apache2 guide</a>: Fallback to this if you couldn't get something to work, or you don't know something.
<h2 align=center>Known issues</h2>
1. https://git.lolcat.ca/lolcat/4get/issues
[^1]: lolcat/4get#40, If having issues with `libsodium`, or `php-apcu`.
[^2]: <a href="https://git.nadeko.net/Fijxu/etc-configs/src/branch/selfhost/nginx/conf.d/4get.conf">git.nadeko.net</a> nadeko.net's 4get instance configuration.

View File

@ -15,7 +15,12 @@ class favicon{
header("Content-Type: image/png");
if(substr_count($url, "/") !== 2){
if(
preg_match(
'/^https?:\/\/[A-Za-z0-9.-]+$/',
$url
) === 0
){
header("X-Error: Only provide the protocol and domain");
$this->defaulticon();

View File

@ -75,6 +75,7 @@ class backend{
break;
case "socks5_hostname":
case "socks5h":
case "socks5a":
curl_setopt($curlproc, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5_HOSTNAME);
curl_setopt($curlproc, CURLOPT_PROXY, $address . ":" . $port);

View File

@ -838,10 +838,10 @@ class frontend{
}
$payload .=
'<a href="https://webcache.googleusercontent.com/search?q=cache:' . $urlencode . '" class="list" target="_BLANK"><img src="/favicon?s=https://google.com" alt="go">Google cache</a>' .
'<a href="https://web.archive.org/web/' . $urlencode . '" class="list" target="_BLANK"><img src="/favicon?s=https://archive.org" alt="ar">Archive.org</a>' .
'<a href="https://archive.ph/newest/' . htmlspecialchars($link) . '" class="list" target="_BLANK"><img src="/favicon?s=https://archive.is" alt="ar">Archive.is</a>' .
'<a href="https://ghostarchive.org/search?term=' . $urlencode . '" class="list" target="_BLANK"><img src="/favicon?s=https://ghostarchive.org" alt="gh">Ghostarchive</a>' .
'<a href="https://arquivo.pt/wayback/' . htmlspecialchars($link) . '" class="list" target="_BLANK"><img src="/favicon?s=https://arquivo.pt" alt="ar">Arquivo.pt</a>' .
'<a href="https://www.bing.com/search?q=url%3A' . $urlencode . '" class="list" target="_BLANK"><img src="/favicon?s=https://bing.com" alt="bi">Bing cache</a>' .
'<a href="https://megalodon.jp/?url=' . $urlencode . '" class="list" target="_BLANK"><img src="/favicon?s=https://megalodon.jp" alt="me">Megalodon</a>' .
'</div>';
@ -939,6 +939,8 @@ class frontend{
"brave" => "Brave",
"yandex" => "Yandex",
"google" => "Google",
//"google_api" => "Google API",
"google_cse" => "Google CSE",
"startpage" => "Startpage",
"qwant" => "Qwant",
"ghostery" => "Ghostery",
@ -963,13 +965,18 @@ class frontend{
"yandex" => "Yandex",
"brave" => "Brave",
"google" => "Google",
"google_cse" => "Google CSE",
"startpage" => "Startpage",
"qwant" => "Qwant",
"yep" => "Yep",
"solofield" => "Solofield",
//"pinterest" => "Pinterest",
"pinterest" => "Pinterest",
"flickr" => "Flickr",
"fivehpx" => "500px",
"vsco" => "VSCO",
"imgur" => "Imgur",
"ftm" => "FindThatMeme"
"ftm" => "FindThatMeme",
//"sankakucomplex" => "SankakuComplex"
]
];
break;

View File

@ -381,6 +381,8 @@ class fuckhtml{
$json_out = null;
$last_char = null;
$keyword_check = null;
for($i=0; $i<strlen($json); $i++){
switch($json[$i]){
@ -396,6 +398,7 @@ class fuckhtml{
$bracket = false;
$is_close_bracket = true;
}else{
if($bracket === false){
@ -429,6 +432,31 @@ class fuckhtml{
$is_close_bracket === false
){
// do keyword check
$keyword_check .= $json[$i];
if(in_array($json[$i], [":", "{"])){
$keyword_check = substr($keyword_check, 0, -1);
if(
preg_match(
'/function|array|return/i',
$keyword_check
)
){
$json_out =
preg_replace(
'/[{"]*' . preg_quote($keyword_check, "/") . '$/',
"",
$json_out
);
}
$keyword_check = null;
}
// here we know we're not iterating over a quoted string
switch($json[$i]){
@ -498,4 +526,85 @@ class fuckhtml{
$string
);
}
public function extract_json($json){
$len = strlen($json);
$array_level = 0;
$object_level = 0;
$in_quote = null;
$start = null;
for($i=0; $i<$len; $i++){
switch($json[$i]){
case "[":
if($in_quote === null){
$array_level++;
if($start === null){
$start = $i;
}
}
break;
case "]":
if($in_quote === null){
$array_level--;
}
break;
case "{":
if($in_quote === null){
$object_level++;
if($start === null){
$start = $i;
}
}
break;
case "}":
if($in_quote === null){
$object_level--;
}
break;
case "\"":
case "'":
if(
$i !== 0 &&
$json[$i - 1] !== "\\"
){
// found a non-escaped quote
if($in_quote === null){
// open quote
$in_quote = $json[$i];
}elseif($in_quote === $json[$i]){
// close quote
$in_quote = null;
}
}
break;
}
if(
$start !== null &&
$array_level === 0 &&
$object_level === 0
){
return substr($json, $start, $i - $start + 1);
break;
}
}
}
}

View File

@ -34,22 +34,46 @@ try{
)
){
if(
!isset($image["query"]) ||
!isset($image["path"]) ||
$image["path"] != "/th"
){
if(!isset($image["path"])){
header("X-Error: Invalid bing image path");
header("X-Error: Missing bing image path");
$proxy->do404();
die();
}
//
// get image ID
// formations:
// https://tse2.mm.bing.net/th/id/OIP.3yLBkUPn8EXA1wlhWP2BHwHaE3
// https://tse2.mm.bing.net/th?id=OIP.3yLBkUPn8EXA1wlhWP2BHwHaE3
//
$id = null;
if(isset($image["query"])){
parse_str($image["query"], $str);
if(!isset($str["id"])){
if(isset($str["id"])){
header("X-Error: Missing bing ID");
$id = $str["id"];
}
}
if($id === null){
$id = explode("/th/id/", $image["path"], 2);
if(count($id) !== 2){
// malformed
return $url;
}
$id = $id[1];
}
if(is_array($id)){
header("X-Error: Missing bing id parameter");
$proxy->do404();
die();
}
@ -63,7 +87,7 @@ try{
case "cover": $req = "&w=207&h=270&p=0&qlt=90"; break;
}
$proxy->stream_linear_image("https://" . $image["host"] . "/th?id=" . urlencode($str["id"]) . $req, "https://www.bing.com");
$proxy->stream_linear_image("https://" . $image["host"] . "/th?id=" . rawurlencode($id) . $req, "https://www.bing.com");
die();
}

View File

@ -210,6 +210,63 @@ class brave{
return $data;
}
private function get_js(){
$script_disc =
$this->fuckhtml
->getElementsByTagName(
"script"
);
$data = null;
foreach($script_disc as &$discs){
if(
preg_match(
'/kit\.start\(/',
$discs["innerHTML"]
)
){
$data =
explode(
"data:",
$discs["innerHTML"],
2
);
if(count($data) !== 2){
throw new Exception("Failed to split up data field");
}
$data = $data[1];
break;
}
}
if($data === null){
throw new Exception("Could not grep JavaScript object");
}
$data =
$this->fuckhtml
->parseJsObject(
$this->fuckhtml
->extract_json(
$data
)
);
if($data === null){
throw new Exception("Failed to decode JavaScript object");
}
return $data;
}
public function web($get){
if($get["npt"]){
@ -293,8 +350,8 @@ class brave{
/*
$handle = fopen("scraper/brave.html", "r");
$html = fread($handle, filesize("scraper/brave.html"));
fclose($handle);
*/
fclose($handle);*/
try{
$html =
@ -346,7 +403,7 @@ class brave{
$nextpage =
$this->fuckhtml
->getElementsByClassName("btn", "a");
->getElementsByClassName("button", "a");
if(count($nextpage) !== 0){
@ -382,45 +439,9 @@ class brave{
}
}
// do some magic
$this->fuckhtml->load($html);
$script_disc =
$this->fuckhtml
->getElementsByTagName(
"script"
);
$grep = [];
foreach($script_disc as $discs){
preg_match(
'/const data ?= ?(\[{.*}]);/',
$discs["innerHTML"],
$grep
);
if(isset($grep[1])){
break;
}
}
if(!isset($grep[1])){
throw new Exception("Could not grep JavaScript object");
}
$data =
$this->fuckhtml
->parseJsObject(
$grep[1]
);
unset($grep);
if($data === null){
throw new Exception("Failed to decode JavaScript object");
}
$data = $this->get_js();
if(
isset($data[2]["data"]["title"]) &&
@ -663,7 +684,10 @@ class brave{
$table["Address"] = $result["location"]["postal_address"]["displayAddress"];
}
if(isset($result["location"]["rating"])){
if(
isset($result["location"]["rating"]) &&
$result["location"]["rating"] != "void 0"
){
$table["Rating"] =
$result["location"]["rating"]["ratingValue"] . "/" .
@ -671,13 +695,19 @@ class brave{
number_format($result["location"]["rating"]["reviewCount"]) . " votes)";
}
if(isset($result["location"]["contact"]["telephone"])){
if(
isset($result["location"]["contact"]["telephone"]) &&
$result["location"]["contact"]["telephone"] != "void 0"
){
$table["Phone number"] =
$result["location"]["contact"]["telephone"];
}
if(isset($result["location"]["price_range"])){
if(
isset($result["location"]["price_range"]) &&
$result["location"]["price_range"] != "void 0"
){
$table["Price"] =
$result["location"]["price_range"];
@ -1160,23 +1190,8 @@ class brave{
$proxy
);
preg_match(
'/const data ?= ?(\[{.*}]);/',
$html,
$json
);
if(!isset($json[1])){
throw new Exception("Failed to grep javascript object");
}
$json = $this->fuckhtml->parseJsObject($json[1], true);
if($json === null){
throw new Exception("Failed to parse javascript object");
}
$this->fuckhtml->load($html);
$json = $this->get_js();
foreach(
$json[1]["data"]["body"]["response"]["news"]["results"]
@ -1258,22 +1273,8 @@ class brave{
$html = fread($handle, filesize("scraper/brave-image.html"));
fclose($handle);*/
preg_match(
'/const data = (\[{.*}\]);/',
$html,
$json
);
if(!isset($json[1])){
throw new Exception("Failed to get data object");
}
$json =
$this->fuckhtml
->parseJsObject(
$json[1]
);
$this->fuckhtml->load($html);
$json = $this->get_js();
foreach(
$json[1]
@ -1403,22 +1404,8 @@ class brave{
$html = fread($handle, filesize("scraper/brave-video.html"));
fclose($handle);*/
preg_match(
'/const data = (\[{.*}\]);/',
$html,
$json
);
if(!isset($json[1])){
throw new Exception("Failed to get data object");
}
$json =
$this->fuckhtml
->parseJsObject(
$json[1]
);
$this->fuckhtml->load($html);
$json = $this->get_js();
foreach(
$json
@ -1790,7 +1777,21 @@ class brave{
$nextpage =
$this->fuckhtml
->getElementsByClassName("btn", "a");
->getElementById(
"pagination",
"div"
);
if($nextpage){
$this->fuckhtml->load($nextpage);
$nextpage =
$this->fuckhtml
->getElementsByClassName(
"button",
"a"
);
if(count($nextpage) !== 0){
@ -1828,6 +1829,7 @@ class brave{
);
}
}
}
return null;
}

File diff suppressed because it is too large Load Diff

262
scraper/fivehpx.php Normal file
View File

@ -0,0 +1,262 @@
<?php
class fivehpx{
public function __construct(){
include "lib/backend.php";
$this->backend = new backend("fivehpx");
include "lib/fuckhtml.php";
$this->fuckhtml = new fuckhtml();
}
public function getfilters($page){
return [
"sort" => [
"display" => "Sort",
"option" => [
"relevance" => "Relevance",
"pulse" => "Pulse",
"newest" => "Newest"
]
]
];
}
private function get($proxy, $url, $get = [], $post_data = null){
$curlproc = curl_init();
if($get !== []){
$get = http_build_query($get);
$url .= "?" . $get;
}
curl_setopt($curlproc, CURLOPT_URL, $url);
curl_setopt($curlproc, CURLOPT_ENCODING, ""); // default encoding
if($post_data === null){
curl_setopt($curlproc, CURLOPT_HTTPHEADER,
["User-Agent: " . config::USER_AGENT,
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip",
"DNT: 1",
"Sec-GPC: 1",
"Connection: keep-alive",
"Upgrade-Insecure-Requests: 1",
"Sec-Fetch-Dest: document",
"Sec-Fetch-Mode: navigate",
"Sec-Fetch-Site: same-origin",
"Sec-Fetch-User: ?1",
"Priority: u=0, i",
"TE: trailers"]
);
}else{
curl_setopt($curlproc, CURLOPT_HTTPHEADER,
["User-Agent: " . config::USER_AGENT,
"Accept: */*",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip",
"Referer: https://500px.com/",
"content-type: application/json",
//"x-csrf-token: undefined",
"x-500px-source: Search",
"Content-Length: " . strlen($post_data),
"Origin: https://500px.com",
"DNT: 1",
"Sec-GPC: 1",
"Connection: keep-alive",
// "Cookie: _pin_unauth, _fbp, _sharedID, _sharedID_cst",
"Sec-Fetch-Dest: empty",
"Sec-Fetch-Mode: cors",
"Sec-Fetch-Site: same-site",
"Priority: u=4",
"TE: trailers"]
);
// set post data
curl_setopt($curlproc, CURLOPT_POST, true);
curl_setopt($curlproc, CURLOPT_POSTFIELDS, $post_data);
}
curl_setopt($curlproc, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curlproc, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($curlproc, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($curlproc, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curlproc, CURLOPT_TIMEOUT, 30);
// http2 bypass
curl_setopt($curlproc, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2_0);
$this->backend->assign_proxy($curlproc, $proxy);
$data = curl_exec($curlproc);
if(curl_errno($curlproc)){
throw new Exception(curl_error($curlproc));
}
curl_close($curlproc);
return $data;
}
public function image($get){
if($get["npt"]){
[$pagination, $proxy] =
$this->backend->get(
$get["npt"], "images"
);
$pagination = json_decode($pagination, true);
$search = $pagination["search"];
}else{
$search = $get["s"];
if(strlen($search) === 0){
throw new Exception("Search term is empty!");
}
$proxy = $this->backend->get_ip();
$pagination = [
"sort" => strtoupper($get["sort"]),
"search" => $search,
"filters" => [],
"nlp" => false,
];
}
try{
$json =
$this->get(
$proxy,
"https://api.500px.com/graphql",
[],
json_encode([
"operationName" => "PhotoSearchPaginationContainerQuery",
"variables" => $pagination,
"query" =>
'query PhotoSearchPaginationContainerQuery(' .
(isset($pagination["cursor"]) ? '$cursor: String, ' : "") .
'$sort: PhotoSort, $search: String!, $filters: [PhotoSearchFilter!], $nlp: Boolean) { ...PhotoSearchPaginationContainer_query_1vzAZD} fragment PhotoSearchPaginationContainer_query_1vzAZD on Query { photoSearch(sort: $sort, first: 100, ' .
(isset($pagination["cursor"]) ? 'after: $cursor, ' : "") .
'search: $search, filters: $filters, nlp: $nlp) { edges { node { id legacyId canonicalPath name description width height images(sizes: [33, 36]) { size url id } } } totalCount pageInfo { endCursor hasNextPage } }}'
])
);
}catch(Exception $error){
throw new Exception("Failed to fetch graphQL object");
}
$json = json_decode($json, true);
if($json === null){
throw new Exception("Failed to decode graphQL object");
}
if(isset($json["errors"][0]["message"])){
throw new Exception("500px returned an API error: " . $json["errors"][0]["message"]);
}
if(!isset($json["data"]["photoSearch"]["edges"])){
throw new Exception("No edges returned by API");
}
$out = [
"status" => "ok",
"npt" => null,
"image" => []
];
foreach($json["data"]["photoSearch"]["edges"] as $image){
$image = $image["node"];
$title =
trim(
$this->fuckhtml
->getTextContent(
$image["name"]
) . ": " .
$this->fuckhtml
->getTextContent(
$image["description"]
)
, " :"
);
$small = $this->image_ratio(600, $image["width"], $image["height"]);
$large = $this->image_ratio(2048, $image["width"], $image["height"]);
$out["image"][] = [
"title" => $title,
"source" => [
[
"url" => $image["images"][1]["url"],
"width" => $large[0],
"height" => $large[1]
],
[
"url" => $image["images"][0]["url"],
"width" => $small[0],
"height" => $small[1]
]
],
"url" => "https://500px.com" . $image["canonicalPath"]
];
}
// get NPT token
if($json["data"]["photoSearch"]["pageInfo"]["hasNextPage"] === true){
$out["npt"] =
$this->backend->store(
json_encode([
"cursor" => $json["data"]["photoSearch"]["pageInfo"]["endCursor"],
"search" => $search,
"sort" => $pagination["sort"],
"filters" => [],
"nlp" => false
]),
"images",
$proxy
);
}
return $out;
}
private function image_ratio($longest_edge, $width, $height){
$ratio = [
$longest_edge / $width,
$longest_edge / $height
];
if($ratio[0] < $ratio[1]){
$ratio = $ratio[0];
}else{
$ratio = $ratio[1];
}
return [
floor($width * $ratio),
floor($height * $ratio)
];
}
}

415
scraper/flickr.php Normal file
View File

@ -0,0 +1,415 @@
<?php
class flickr{
const req_web = 0;
const req_xhr = 1;
public function __construct(){
include "lib/backend.php";
$this->backend = new backend("flickr");
include "lib/fuckhtml.php";
$this->fuckhtml = new fuckhtml();
}
public function getfilters($page){
return [
"nsfw" => [
"display" => "NSFW",
"option" => [
"yes" => "Yes",
"maybe" => "Maybe",
"no" => "No",
]
],
"sort" => [
"display" => "Sort by",
"option" => [
"relevance" => "Relevance",
"date-posted-desc" => "Newest uploads",
"date-posted-asc" => "Oldest uploads",
"date-taken-desc" => "Newest taken",
"date-taken-asc" => "Oldest taken",
"interestingness-desc" => "Interesting"
]
],
"color" => [
"display" => "Color",
"option" => [
"any" => "Any color",
// color_codes=
"0" => "Red",
"1" => "Brown",
"2" => "Orange",
"b" => "Pink",
"4" => "Yellow",
"3" => "Golden",
"5" => "Lime",
"6" => "Green",
"7" => "Sky blue",
"8" => "Blue",
"9" => "Purple",
"a" => "Hot pink",
"c" => "White",
"d" => "Gray",
"e" => "Black",
// styles= override
"blackandwhite" => "Black & white",
]
],
"style" => [ // styles=
"display" => "Style",
"option" => [
"any" => "Any style",
"depthoffield" => "Depth of field",
"minimalism" => "Minimalism",
"pattern" => "Patterns"
]
],
"license" => [
"display" => "License",
"option" => [
"any" => "Any license",
"1,2,3,4,5,6,9,11,12,13,14,15,16" => "All creative commons",
"4,5,6,9,10,11,12,13" => "Commercial use allowed",
"1,2,4,5,9,10,11,12,14,15" => "Modifications allowed",
"4,5,9,10,11,12" => "Commercial use & mods allowed",
"7,9,10" => "No known copyright restrictions",
"8" => "U.S Government works"
]
]
];
}
private function get($proxy, $url, $get = [], $reqtype){
$curlproc = curl_init();
if($get !== []){
$get = http_build_query($get);
$url .= "?" . $get;
}
curl_setopt($curlproc, CURLOPT_URL, $url);
curl_setopt($curlproc, CURLOPT_ENCODING, ""); // default encoding
if($reqtype === flickr::req_web){
curl_setopt($curlproc, CURLOPT_HTTPHEADER,
["User-Agent: " . config::USER_AGENT,
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip",
"DNT: 1",
"Sec-GPC: 1",
"Connection: keep-alive",
"Upgrade-Insecure-Requests: 1",
"Sec-Fetch-Dest: document",
"Sec-Fetch-Mode: navigate",
"Sec-Fetch-Site: same-origin",
"Sec-Fetch-User: ?1",
"Priority: u=0, i",
"TE: trailers"]
);
}else{
curl_setopt($curlproc, CURLOPT_HTTPHEADER,
["User-Agent: " . config::USER_AGENT,
"Accept: */*",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip",
"Origin: https://www.flickr.com",
"DNT: 1",
"Sec-GPC: 1",
"Connection: keep-alive",
"Referer: https://www.flickr.com/",
// Cookie:
"Sec-Fetch-Dest: empty",
"Sec-Fetch-Mode: cors",
"Sec-Fetch-Site: same-site",
"TE: trailers"]
);
}
curl_setopt($curlproc, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curlproc, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($curlproc, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($curlproc, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curlproc, CURLOPT_TIMEOUT, 30);
// http2 bypass
curl_setopt($curlproc, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2_0);
$this->backend->assign_proxy($curlproc, $proxy);
$data = curl_exec($curlproc);
if(curl_errno($curlproc)){
throw new Exception(curl_error($curlproc));
}
curl_close($curlproc);
return $data;
}
public function image($get){
if($get["npt"]){
[$filters, $proxy] =
$this->backend->get(
$get["npt"], "images"
);
$filters = json_decode($filters, true);
// Workaround for the future, if flickr deprecates &page argument on html page
/*
try{
$json =
$this->get(
$proxy,
"https://api.flickr.com/services/rest",
[
"sort" => $data["sort"],
"parse_tags" => 1,
// url_s,url_n,url_w,url_m,url_z,url_c,url_l,url_h,url_k,url_3k,url_4k,url_5k,url_6k,url_o
"extras" => "can_comment,can_print,count_comments,count_faves,description,isfavorite,license,media,needs_interstitial,owner_name,path_alias,realname,rotation,url_sq,url_q,url_t,url_s,url_n,url_w,url_m,url_z,url_c,url_l",
"per_page" => 100,
"page" => $data["page"],
"lang" => "en-US",
"text" => $data["search"],
"viewerNSID" => "",
"method" => "flickr.photos.search",
"csrf" => "",
"api_key" => $data["api_key"],
"format" => "json",
"hermes" => 1,
"hermesClient" => 1,
"reqId" => $data["reqId"],
"nojsoncallback" => 1
]
);
}catch(Exception $error){
throw new Exception("Failed to fetch JSON");
}*/
}else{
if(strlen($get["s"]) === 0){
throw new Exception("Search term is empty!");
}
$proxy = $this->backend->get_ip();
// compute filters
$filters = [
"page" => 1,
"sort" => $get["sort"]
];
if($get["style"] != "any"){
$filters["styles"] = $get["style"];
}
if($get["color"] != "any"){
if($get["color"] != "blackandwhite"){
$filters["color_codes"] = $get["color"];
}else{
$filters["styles"] = "blackandwhite";
}
}
if($get["license"] != "any"){
$filters["license"] = $get["license"];
}
switch($get["nsfw"]){
case "yes": $filters["safe_search"] = 0; break;
case "maybe": $filters["safe_search"] = 2; break;
case "no": $filters["safe_search"] = 1; break;
}
}
$get_params = [
"text" => $get["s"],
"per_page" => 50,
// scrape highest resolution
"extras" => "url_s,url_n,url_w,url_m,url_z,url_c,url_l,url_h,url_k,url_3k,url_4k,url_5k,url_6k,url_o",
"view_all" => 1
];
$get_params = array_merge($get_params, $filters);
$html =
$this->get(
$proxy,
"https://www.flickr.com/search/",
$get_params,
flickr::req_web
);
// @TODO
// get api_key and reqId, if flickr deprecates &page
$this->fuckhtml->load($html);
//
// get response JSON
//
$scripts =
$this->fuckhtml
->getElementsByClassName(
"modelExport",
"script"
);
$found = false;
foreach($scripts as $script){
$json =
preg_split(
'/modelExport: ?/',
$script["innerHTML"],
2
);
if(count($json) !== 0){
$found = true;
$json = $json[1];
break;
}
}
if($found === false){
throw new Exception("Failed to grep JSON");
}
$json =
json_decode(
$this->fuckhtml
->extract_json(
$json
),
true
);
if($json === null){
throw new Exception("Failed to decode JSON");
}
$out = [
"status" => "ok",
"npt" => null,
"image" => []
];
if(!isset($json["main"]["search-photos-lite-models"][0]["data"]["photos"]["data"]["_data"])){
throw new Exception("Failed to access data object");
}
foreach($json["main"]["search-photos-lite-models"][0]["data"]["photos"]["data"]["_data"] as $image){
if(!isset($image["data"])){
// flickr likes to gives us empty array objects
continue;
}
$image = $image["data"];
$title = [];
if(isset($image["title"])){
$title[] =
$this->fuckhtml
->getTextContent(
$image["title"]
);
}
if(isset($image["description"])){
$title[] =
$this->fuckhtml
->getTextContent(
str_replace(
"\n",
" ",
$image["description"]
)
);
}
$title = implode(": ", $title);
$sources = array_values($image["sizes"]["data"]);
$suitable_sizes = ["n", "m", "w", "s"];
$thumb = &$sources[0]["data"];
foreach($suitable_sizes as $testing_size){
if(isset($image["sizes"]["data"][$testing_size])){
$thumb = &$image["sizes"]["data"][$testing_size]["data"];
break;
}
}
$og = &$sources[count($sources) - 1]["data"];
$out["image"][] = [
"title" => $title,
"source" => [
[
"url" => "https:" . $og["displayUrl"],
"width" => (int)$og["width"],
"height" => (int)$og["height"]
],
[
"url" => "https:" . $thumb["displayUrl"],
"width" => (int)$thumb["width"],
"height" => (int)$thumb["height"]
]
],
"url" => "https://www.flickr.com/photos/" . $image["ownerNsid"] . "/" . $image["id"] . "/"
];
}
$total_items = (int)$json["main"]["search-photos-lite-models"][0]["data"]["photos"]["data"]["totalItems"];
if(($filters["page"]) * 50 < $total_items){
$filters["page"]++;
$out["npt"] =
$this->backend->store(
json_encode($filters),
"images",
$proxy
);
}
return $out;
}
}

View File

@ -136,7 +136,7 @@ class ftm{
"source" => [
[
"url" =>
"https://findthatmeme.us-southeast-1.linodeobjects.com/" .
"https://s3.thehackerblog.com/findthatmeme/" .
$thumb,
"width" => null,
"height" => null

File diff suppressed because it is too large Load Diff

1054
scraper/google_cse.php Normal file

File diff suppressed because it is too large Load Diff

View File

@ -220,13 +220,14 @@ class marginalia{
"related" => []
];
// API scraper
if(config::MARGINALIA_API_KEY !== null){
try{
$json =
$this->get(
$this->backend->get_ip(), // no nextpage
"https://api.marginalia.nu/" . config::MARGINALIA_API_KEY . "/search/" . urlencode($search),
"https://api.marginalia-search.com/" . config::MARGINALIA_API_KEY . "/search/" . urlencode($search),
[
"count" => 20
]
@ -263,7 +264,29 @@ class marginalia{
return $out;
}
// no more cloudflare!! Parse html by default
// HTML parser
$proxy = $this->backend->get_ip();
if($get["npt"]){
[$params, $proxy] =
$this->backend->get(
$get["npt"],
"web"
);
try{
$html =
$this->get(
$proxy,
"https://old-search.marginalia.nu/search?" . $params
);
}catch(Exception $error){
throw new Exception("Failed to get HTML");
}
}else{
$params = [
"query" => $search
];
@ -284,14 +307,15 @@ class marginalia{
try{
$html =
$this->get(
$this->backend->get_ip(),
"https://search.marginalia.nu/search",
$proxy,
"https://old-search.marginalia.nu/search",
$params
);
}catch(Exception $error){
throw new Exception("Failed to get HTML");
}
}
$this->fuckhtml->load($html);
@ -387,6 +411,65 @@ class marginalia{
];
}
// get next page
$this->fuckhtml->load($html);
$pagination =
$this->fuckhtml
->getElementsByAttributeValue(
"aria-label",
"pagination",
"nav"
);
if(count($pagination) === 0){
// no pagination
return $out;
}
$this->fuckhtml->load($pagination[0]);
$pages =
$this->fuckhtml
->getElementsByClassName(
"page-link",
"a"
);
$found_current_page = false;
foreach($pages as $page){
if(
stripos(
$page["attributes"]["class"],
"active"
) !== false
){
$found_current_page = true;
continue;
}
if($found_current_page){
// we found current page index, and we iterated over
// the next page <a>
$out["npt"] =
$this->backend->store(
parse_url(
$page["attributes"]["href"],
PHP_URL_QUERY
),
"web",
$proxy
);
break;
}
}
return $out;
}
}

View File

@ -701,9 +701,11 @@ class mojeek{
if(count($thumb) === 2){
$answer["thumb"] =
urldecode(
$this->fuckhtml
->getTextContent(
$thumb[1]
)
);
}
}

View File

@ -13,31 +13,104 @@ class pinterest{
return [];
}
private function get($proxy, $url, $get = []){
private function get($proxy, $url, $get = [], &$cookies, $header_data_post = null){
$curlproc = curl_init();
if($header_data_post === null){
// handling GET
// extract cookies
$cookies_tmp = [];
curl_setopt($curlproc, CURLOPT_HEADERFUNCTION, function($curlproc, $header) use (&$cookies_tmp){
$length = strlen($header);
$header = explode(":", $header, 2);
if(trim(strtolower($header[0])) == "set-cookie"){
$cookie_tmp = explode("=", trim($header[1]), 2);
$cookies_tmp[trim($cookie_tmp[0])] =
explode(";", $cookie_tmp[1], 2)[0];
}
return $length;
});
curl_setopt($curlproc, CURLOPT_HTTPHEADER,
["User-Agent: " . config::USER_AGENT,
"Accept: application/json, text/javascript, */*, q=0.01",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip",
"Referer: https://ca.pinterest.com/",
"X-Requested-With: XMLHttpRequest",
"X-APP-VERSION: 78f8764",
"X-Pinterest-AppState: active",
"X-Pinterest-Source-Url: /",
"X-Pinterest-PWS-Handler: www/index.js",
"screen-dpr: 1",
"is-preload-enabled: 1",
"DNT: 1",
"Sec-GPC: 1",
"Sec-Fetch-Dest: empty",
"Sec-Fetch-Mode: cors",
"Sec-Fetch-Site: same-origin",
"Connection: keep-alive",
"Alt-Used: ca.pinterest.com",
"Priority: u=0",
"TE: trailers"]
);
if($get !== []){
$get = http_build_query($get);
$url .= "?" . $get;
}
}else{
// handling POST (pagination)
$get = http_build_query($get);
curl_setopt($curlproc, CURLOPT_HTTPHEADER,
["User-Agent: " . config::USER_AGENT,
"Accept: application/json, text/javascript, */*, q=0.01",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip",
"Content-Type: application/x-www-form-urlencoded",
"Content-Length: " . strlen($get),
"Referer: https://ca.pinterest.com/",
"X-Requested-With: XMLHttpRequest",
"X-APP-VERSION: 78f8764",
"X-CSRFToken: " . $cookies["csrf"],
"X-Pinterest-AppState: active",
"X-Pinterest-Source-Url: /search/pins/?rs=ac&len=2&q=" . urlencode($header_data_post) . "&eq=" . urlencode($header_data_post),
"X-Pinterest-PWS-Handler: www/search/[scope].js",
"screen-dpr: 1",
"is-preload-enabled: 1",
"Origin: https://ca.pinterest.com",
"DNT: 1",
"Sec-GPC: 1",
"Sec-Fetch-Dest: empty",
"Sec-Fetch-Mode: cors",
"Sec-Fetch-Site: same-origin",
"Connection: keep-alive",
"Alt-Used: ca.pinterest.com",
"Cookie: " . $cookies["cookie"],
"TE: trailers"]
);
curl_setopt($curlproc, CURLOPT_POST, true);
curl_setopt($curlproc, CURLOPT_POSTFIELDS, $get);
}
curl_setopt($curlproc, CURLOPT_URL, $url);
curl_setopt($curlproc, CURLOPT_ENCODING, ""); // default encoding
curl_setopt($curlproc, CURLOPT_HTTPHEADER,
["User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:107.0) Gecko/20100101 Firefox/110.0",
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip",
"DNT: 1",
"Connection: keep-alive",
"Upgrade-Insecure-Requests: 1",
"Sec-Fetch-Dest: document",
"Sec-Fetch-Mode: navigate",
"Sec-Fetch-Site: none",
"Sec-Fetch-User: ?1"]
);
// http2 bypass
curl_setopt($curlproc, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2_0);
curl_setopt($curlproc, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curlproc, CURLOPT_SSL_VERIFYHOST, 2);
@ -54,6 +127,26 @@ class pinterest{
throw new Exception(curl_error($curlproc));
}
if($header_data_post === null){
if(!isset($cookies_tmp["csrftoken"])){
throw new Exception("Failed to grep CSRF token");
}
$cookies = "";
foreach($cookies_tmp as $cookie_name => $cookie_value){
$cookies .= $cookie_name . "=" . $cookie_value . "; ";
}
$cookies = [
"csrf" => $cookies_tmp["csrftoken"],
"cookie" => rtrim($cookies, " ;")
];
}
curl_close($curlproc);
return $data;
}
@ -62,17 +155,68 @@ class pinterest{
if($get["npt"]){
// @TODO
// post data for next page
$data = [
"source_url" => "/search/pins/?q=" . urlencode($search) . "&rs=typed",
"data" =>
json_encode(
[
// {"options":{"applied_filters":null,"appliedProductFilters":"---","article":null,"auto_correction_disabled":false,"corpus":null,"customized_rerank_type":null,"domains":null,"filters":null,"journey_depth":null,"page_size":null,"price_max":null,"price_min":null,"query_pin_sigs":null,"query":"higurashi","redux_normalize_feed":true,"rs":"typed","scope":"pins","selected_one_bar_modules":null,"source_id":null,"source_module_id":null,"top_pin_id":null,"bookmarks":["Y2JVSG81V2sxcmNHRlpWM1J5VFVad1ZsWlVRbXhpVmtreVZsZHpOV0pIU2tkV2FscFhVbXhhVkZreU1WSmtNREZWVjIxR1RrMXNTbEJXYlhSaFVtMVdjMVZ1U2xaaWEzQnpXVlJPVTJWV1pISlhhM1JYVm10V05sVldVbE5XVjBwMVVXMUdWVll6VFhoVWJYaFhWMVp3Ums1V1RsTmlSbGt5Vm10YWFtVkdWbkpOU0dSUFZsZG9XRmxzWkc5VlZscHlWbGhrYkdKR1NubFdWelZQWVVaYWRHVkVRbFppUmtwVVZrUktWMlJIVWtWV2JHaHBVakZLU0Zkc1pEUmtNVnBZVW10b2FsSXdXbkJXYlRWRFpHeGFSMWRzVG1oaGVrWllXV3RvVTFVeFpFaFZiRUpoVm5wRk1GbHFSbXRYVjA1R1YyczFWMVpHV2pSWFZtaDNVakZrY2sxWVRsaGlhM0JXV1ZSR1MyRkdiRlZTYm1SVVVteHdXbGxWVlRGVk1VbDVWRmhrVjAxdVVuWlVhMXBTWlVaT2MxcEhSbE5TTWswMVdtdGFWMU5YU2paVmJYaFRUVmhDUjFZeU5YZFVNVkY0VjJ0b1ZXRnJOVlpVVmxwTFVURndXR042VmxOV2ExcGFXVlZWTlZVeFNYZE5WRTVYVWtWYVZGWkhNVTlXTVU1WllVWk9hR1ZyV2s1WFZ6QXhZakpPVjFWWWFHRlNWbkJRVm14U1IwMUdXWGxOVkVKVlRWWnNORll5TURWV1YwVjVWV3hDV21FeGNETmFSVnByVjFkS1IyTkhhR2xYUjJkM1ZtdGFhMlF4VVhsVGJGcE9Wa1p3YjFwWGVFdFZWbFp4VW14YWJGWnRVbHBaTUdoTFZHMUtTR1ZJYUZkV2VrWjJWMVphU21ReVJYcGpSbFpwVW10d1RGZHJVa0pPVms1SFZHNVNUbFl3V2xoVmJYUldaVVpaZUZremFGUk5hM0JYVkZaYVYyRkZNSGxWYkVKYVlrWlZlRnBGV210WFIwNUpVMnMxVTFaR1dscFdWekI0VFVaV1IxTllaR3BUUlhCb1dWUkdWbVZHVm5SbFJuQnNZbFpKTWxSVlVYaFBSVGxGV1hwR1QyVnJSVEZVVlZKT1RrVXhSVkpVUWs5bGJFVXhWRmhzZDFOR1ZsWmtNMFp0VWpGYWIxZFhjRXBsUlRGSVZWaHdUbFl4YTNoVVZWSnFUVVUxV0ZadGFFOVNSVnB6Vkd0a1drMUdiRFpUVkVaT1pXMWplRmRzVWxkaFJuQllWVlJTVDJWdFRqWlVNVkpTWlZad2NWcEhkRTlsYTFwMFZGVlNhMkpWTVZWVFZFcE9Wa1pzTmxkWE1WSk9WVEYwVlcweFVGWXdXVFJXUjNSWFYwZGFRbEJVTVRoUFJHTXhUbnBCTlUxRVRUUk5SRVV3VG5wUk5VMTVjRWhWVlhkeFprUlZlRTlFVVRKWlZHc3lUMWRSTWsxVVVUSk9iVnBvV1RKWmVrNTZXWGhPTWs1cFQwUkZNVTlFVm1sTlZGcHBUV3BTYTFsWFRtcE9SR015VG1wVk5GbHFaR2haVjFacldWUmFiVmxxWkdoYVZGWnFUa1JXT0ZSclZsaG1RVDA5fFVIbzVhRkpYZUc1WFYyUlpWVEpHYkdGNk1XWk5ha1ptVFZSR09FOUVZekZPZWtFMVRVUk5ORTFFUlRCT2VsRTFUWGx3U0ZWVmQzRm1SMWw1VFZSUk1WbDZUVEJhUjFGNVQxZFNhVnB0VlRGT1JFVXdXVlJuZVU1cVRUUk5hbU40VDBSSk1VNXFWVEZOYlZwcVdsUnJlRTFFVVhwWmVsVjNXbXBvYkU1dFJYbE9ha0Y2VDFSSk5VMTZWVEJaYWtJNFZHdFdXR1pCUFQwPXxOb25lfDg3NTcwOTAzODAxNDc0OTMqR1FMKnwzMjM3YjM3ZGNhMGU3YjYyYzYzYzAyZGJkNGU1MjdlNzMyMTExMTNlMmUyMzEyOWM2MDAzYmU1ZTlmZjkwYjAwfE5FV3w="]},"context":{}}
]
[$data, $proxy] =
$this->backend->get(
$get["npt"], "images"
);
];
$data = json_decode($data, true);
$search = $data["q"];
$cookies = $data["cookies"];
try{
$json =
$this->get(
$proxy,
"https://ca.pinterest.com/resource/BaseSearchResource/get/",
[
"source_url" => "/search/pins/?q=" . urlencode($search) . "&rs=typed",
"data" => json_encode(
[
"options" => [
"applied_unified_filters" => null,
"appliedProductFilters" => "---",
"article" => null,
"auto_correction_disabled" => false,
"corpus" => null,
"customized_rerank_type" => null,
"domains" => null,
"dynamicPageSizeExpGroup" => null,
"filters" => null,
"journey_depth" => null,
"page_size" => null,
"price_max" => null,
"price_min" => null,
"query_pin_sigs" => null,
"query" => $data["q"],
"redux_normalize_feed" => true,
"request_params" => null,
"rs" => "typed",
"scope" => "pins",
"selected_one_bar_modules" => null,
"source_id" => null,
"source_module_id" => null,
"source_url" => "/search/pins/?q=" . urlencode($search) . "&rs=typed",
"top_pin_id" => null,
"top_pin_ids" => null,
"bookmarks" => [
$data["bookmark"]
]
],
"context" => []
],
JSON_UNESCAPED_SLASHES
)
],
$cookies,
$search
);
}catch(Exception $error){
throw new Exception("Failed to fetch JSON");
}
}else{
@ -82,26 +226,44 @@ class pinterest{
throw new Exception("Search term is empty!");
}
// https://ca.pinterest.com/resource/BaseSearchResource/get/?source_url=%2Fsearch%2Fpins%2F%3Feq%3Dhigurashi%26etslf%3D5966%26len%3D2%26q%3Dhigurashi%2520when%2520they%2520cry%26rs%3Dac&data=%7B%22options%22%3A%7B%22applied_unified_filters%22%3Anull%2C%22appliedProductFilters%22%3A%22---%22%2C%22article%22%3Anull%2C%22auto_correction_disabled%22%3Afalse%2C%22corpus%22%3Anull%2C%22customized_rerank_type%22%3Anull%2C%22domains%22%3Anull%2C%22dynamicPageSizeExpGroup%22%3Anull%2C%22filters%22%3Anull%2C%22journey_depth%22%3Anull%2C%22page_size%22%3Anull%2C%22price_max%22%3Anull%2C%22price_min%22%3Anull%2C%22query_pin_sigs%22%3Anull%2C%22query%22%3A%22higurashi%20when%20they%20cry%22%2C%22redux_normalize_feed%22%3Atrue%2C%22request_params%22%3Anull%2C%22rs%22%3A%22ac%22%2C%22scope%22%3A%22pins%22%2C%22selected_one_bar_modules%22%3Anull%2C%22source_id%22%3Anull%2C%22source_module_id%22%3Anull%2C%22source_url%22%3A%22%2Fsearch%2Fpins%2F%3Feq%3Dhigurashi%26etslf%3D5966%26len%3D2%26q%3Dhigurashi%2520when%2520they%2520cry%26rs%3Dac%22%2C%22top_pin_id%22%3Anull%2C%22top_pin_ids%22%3Anull%7D%2C%22context%22%3A%7B%7D%7D&_=1736116313987
// source_url=%2Fsearch%2Fpins%2F%3Feq%3Dhigurashi%26etslf%3D5966%26len%3D2%26q%3Dhigurashi%2520when%2520they%2520cry%26rs%3Dac
// &data=%7B%22options%22%3A%7B%22applied_unified_filters%22%3Anull%2C%22appliedProductFilters%22%3A%22---%22%2C%22article%22%3Anull%2C%22auto_correction_disabled%22%3Afalse%2C%22corpus%22%3Anull%2C%22customized_rerank_type%22%3Anull%2C%22domains%22%3Anull%2C%22dynamicPageSizeExpGroup%22%3Anull%2C%22filters%22%3Anull%2C%22journey_depth%22%3Anull%2C%22page_size%22%3Anull%2C%22price_max%22%3Anull%2C%22price_min%22%3Anull%2C%22query_pin_sigs%22%3Anull%2C%22query%22%3A%22higurashi%20when%20they%20cry%22%2C%22redux_normalize_feed%22%3Atrue%2C%22request_params%22%3Anull%2C%22rs%22%3A%22ac%22%2C%22scope%22%3A%22pins%22%2C%22selected_one_bar_modules%22%3Anull%2C%22source_id%22%3Anull%2C%22source_module_id%22%3Anull%2C%22source_url%22%3A%22%2Fsearch%2Fpins%2F%3Feq%3Dhigurashi%26etslf%3D5966%26len%3D2%26q%3Dhigurashi%2520when%2520they%2520cry%26rs%3Dac%22%2C%22top_pin_id%22%3Anull%2C%22top_pin_ids%22%3Anull%7D%2C%22context%22%3A%7B%7D%7D
// &_=1736116313987
$source_url = "/search/pins/?q=" . urlencode($search) . "&rs=" . urlencode($search);
$filter = [
"source_url" => "/search/pins/?q=" . urlencode($search),
"source_url" => $source_url,
"rs" => "typed",
"data" =>
json_encode(
[
"options" => [
"article" => null,
"applied_filters" => null,
"applied_unified_filters" => null,
"appliedProductFilters" => "---",
"auto_correction_disabled" => false,
"article" => null,
"corpus" => null,
"customized_rerank_type" => null,
"domains" => null,
"dynamicPageSizeExpGroup" => null,
"filters" => null,
"query" => $search,
"journey_depth" => null,
"page_size" => null,
"price_max" => null,
"price_min" => null,
"query_pin_sigs" => null,
"query" => $search,
"redux_normalize_feed" => true,
"rs" => "typed",
"request_params" => null,
"rs" => "ac",
"scope" => "pins", // pins, boards, videos,
"source_id" => null
"selected_one_bar_modules" => null,
"source_id" => null,
"source_module_id" => null,
"source_url" => $source_url,
"top_pin_id" => null,
"top_pin_ids" => null
],
"context" => []
]
@ -110,23 +272,25 @@ class pinterest{
];
$proxy = $this->backend->get_ip();
}
$cookies = [];
try{
$json =
json_decode(
$this->get(
$proxy,
"https://www.pinterest.ca/resource/BaseSearchResource/get/",
$filter
),
true
"https://ca.pinterest.com/resource/BaseSearchResource/get/",
$filter,
$cookies,
null
);
}catch(Exception $error){
throw new Exception("Failed to fetch JSON");
}
}
$json = json_decode($json, true);
if($json === null){
@ -139,6 +303,60 @@ class pinterest{
"image" => []
];
if(
!isset(
$json["resource_response"]
["status"]
)
){
throw new Exception("Unknown API failure");
}
if($json["resource_response"]["status"] != "success"){
$status = "Got non-OK response: " . $json["resource_response"]["status"];
if(
isset(
$json["resource_response"]["message"]
)
){
$status .= " - " . $json["resource_response"]["message"];
}
throw new Exception($status);
}
if(
isset(
$json["resource_response"]["sensitivity"]
["notices"][0]["description"]["text"]
)
){
throw new Exception(
"Pinterest returned a notice: " .
$json["resource_response"]["sensitivity"]["notices"][0]["description"]["text"]
);
}
// get NPT
if(isset($json["resource_response"]["bookmark"])){
$out["npt"] =
$this->backend->store(
json_encode([
"q" => $search,
"bookmark" => $json["resource_response"]["bookmark"],
"cookies" => $cookies
]),
"images",
$proxy
);
}
foreach(
$json
["resource_response"]
@ -150,6 +368,7 @@ class pinterest{
switch($item["type"]){
case "pin":
case "board":
/*
Handle image object
@ -206,42 +425,15 @@ class pinterest{
"height" => (int)$thumb["height"]
]
],
"url" => "https://www.pinterest.com/pin/" . $item["id"]
];
break;
case "board":
if(isset($item["cover_pin"]["image_url"])){
$image = [
"url" => $item["cover_pin"]["image_url"],
"width" => (int)$item["cover_pin"]["size"][0],
"height" => (int)$item["cover_pin"]["size"][1]
];
}elseif(isset($item["image_cover_url_hd"])){
/*
$image = [
"url" =>
"width" => null,
"height" => null
];*/
}
$item["link"] === null ?
"https://ca.pinterest.com/pin/" . $item["id"] :
$item["link"]
];
break;
}
}
return $out;
}
private function getfullresimage($image, $has_og){
$has_og = $has_og ? "1200x" : "originals";
return
preg_replace(
'/https:\/\/i\.pinimg\.com\/[^\/]+\//',
"https://i.pinimg.com/" . $has_og . "/",
$image
);
}
}

View File

@ -410,10 +410,7 @@ class qwant{
"thumb" =>
$answer["data"]["result"]["thumbnail"]["landscape"] == null ?
null :
$this->unshitimage(
$answer["data"]["result"]["thumbnail"]["landscape"],
false
),
$this->unshitimage($answer["data"]["result"]["thumbnail"]["landscape"]),
"table" => [],
"sublink" => []
];
@ -770,7 +767,7 @@ class qwant{
}else{
$thumb = [
"url" => $this->unshitimage($video["thumbnail"], false),
"url" => $this->unshitimage($video["thumbnail"]),
"ratio" => "16:9"
];
}
@ -870,7 +867,7 @@ class qwant{
}else{
$thumb = [
"url" => $this->unshitimage($news["media"][0]["pict_big"]["url"], false),
"url" => $this->unshitimage($news["media"][0]["pict_big"]["url"]),
"ratio" => "16:9"
];
}
@ -920,18 +917,77 @@ class qwant{
return trim($text, ". ");
}
private function unshitimage($url, $is_bing = true){
private function unshitimage($url){
// https://s1.qwant.com/thumbr/0x0/8/d/f6de4deb2c2b12f55d8bdcaae576f9f62fd58a05ec0feeac117b354d1bf5c2/th.jpg?u=https%3A%2F%2Fwww.bing.com%2Fth%3Fid%3DOIP.vvDWsagzxjoKKP_rOqhwrQAAAA%26w%3D160%26h%3D160%26c%3D7%26pid%3D5.1&q=0&b=1&p=0&a=0
parse_str(parse_url($url)["query"], $parts);
// https://s2.qwant.com/thumbr/474x289/7/f/412d13b3fe3a03eb2b89633c8e88b609b7d0b93cdd9a5e52db3c663e41e65e/th.jpg?u=https%3A%2F%2Ftse.mm.bing.net%2Fth%3Fid%3DOIP.9Tm_Eo6m7V7ltN19mxduDgHaEh%26pid%3DApi&q=0&b=1&p=0&a=0
if($is_bing){
$parse = parse_url($parts["u"]);
parse_str($parse["query"], $parts);
$image = parse_url($url);
return "https://" . $parse["host"] . "/th?id=" . urlencode($parts["id"]);
if(
!isset($image["host"]) ||
!isset($image["query"])
){
// cant do anything
return $url;
}
return $parts["u"];
$id = null;
if(
preg_match(
'/s[0-9]+\.qwant\.com$/',
$image["host"]
)
){
parse_str($image["query"], $str);
// we're being served a proxy URL
if(isset($str["u"])){
$bing_url = $str["u"];
}else{
// give up
return $url;
}
}
// parse bing URL
$id = null;
$image = parse_url($bing_url);
if(isset($image["query"])){
parse_str($image["query"], $str);
if(isset($str["id"])){
$id = $str["id"];
}
}
if($id === null){
$id = explode("/th/id/", $image["path"], 2);
if(count($id) !== 2){
// malformed
return $url;
}
$id = $id[1];
}
if(is_array($id)){
// fuck off, let proxy.php deal with it
return $url;
}
return "https://" . $image["host"] . "/th?id=" . rawurlencode($id);
}
}

257
scraper/vsco.php Normal file
View File

@ -0,0 +1,257 @@
<?php
class vsco{
public function __construct(){
include "lib/backend.php";
$this->backend = new backend("vsco");
}
public function getfilters($page){
return [];
}
private function get($proxy, $url, $get = [], $bearer = null){
$curlproc = curl_init();
if($get !== []){
$get_tmp = http_build_query($get);
$url .= "?" . $get_tmp;
}
curl_setopt($curlproc, CURLOPT_URL, $url);
curl_setopt($curlproc, CURLOPT_ENCODING, ""); // default encoding
if($bearer === null){
curl_setopt($curlproc, CURLOPT_HTTPHEADER,
["User-Agent: " . config::USER_AGENT,
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip",
"DNT: 1",
"Sec-GPC: 1",
"Connection: keep-alive",
"Upgrade-Insecure-Requests: 1",
"Sec-Fetch-Dest: document",
"Sec-Fetch-Mode: navigate",
"Sec-Fetch-Site: same-origin",
"Sec-Fetch-User: ?1",
"Priority: u=0, i",
"TE: trailers"]
);
}else{
curl_setopt($curlproc, CURLOPT_HTTPHEADER,
["User-Agent: " . config::USER_AGENT,
"Accept: */*",
"Accept-Language: en-US",
"Accept-Encoding: gzip",
"Referer: https://vsco.co/search/images/" . urlencode($get["query"]),
"authorization: Bearer " . $bearer,
"content-type: application/json",
"x-client-build: 1",
"x-client-platform: web",
"DNT: 1",
"Sec-GPC: 1",
"Connection: keep-alive",
"Sec-Fetch-Dest: empty",
"Sec-Fetch-Mode: cors",
"Sec-Fetch-Site: same-origin",
"Priority: u=0",
"TE: trailers"]
);
}
curl_setopt($curlproc, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curlproc, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($curlproc, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($curlproc, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curlproc, CURLOPT_TIMEOUT, 30);
// http2 bypass
curl_setopt($curlproc, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2_0);
$this->backend->assign_proxy($curlproc, $proxy);
$data = curl_exec($curlproc);
if(curl_errno($curlproc)){
throw new Exception(curl_error($curlproc));
}
curl_close($curlproc);
return $data;
}
public function image($get){
if($get["npt"]){
[$data, $proxy] =
$this->backend->get(
$get["npt"], "images"
);
$data = json_decode($data, true);
}else{
$search = $get["s"];
if(strlen($search) === 0){
throw new Exception("Search term is empty!");
}
$proxy = $this->backend->get_ip();
// get bearer token
try{
$html =
$this->get(
$proxy,
"https://vsco.co/feed"
);
}catch(Exception $error){
throw new Exception("Failed to fetch feed page");
}
preg_match(
'/"tkn":"([A-z0-9]+)"/',
$html,
$bearer
);
if(!isset($bearer[1])){
throw new Exception("Failed to grep bearer token");
}
$data = [
"pagination" => [
"query" => $search,
"page" => 0,
"size" => 100
],
"bearer" => $bearer[1]
];
}
try{
$json =
$this->get(
$proxy,
"https://vsco.co/api/2.0/search/images",
$data["pagination"],
$data["bearer"]
);
}catch(Exception $error){
throw new Exception("Failed to fetch JSON");
}
$json = json_decode($json, true);
if($json === null){
throw new Exception("Failed to decode JSON");
}
$out = [
"status" => "ok",
"npt" => null,
"image" => []
];
if(!isset($json["results"])){
throw new Exception("Failed to access results object");
}
foreach($json["results"] as $image){
$image_domain = parse_url("https://" . $image["responsive_url"], PHP_URL_HOST);
$thumbnail = explode($image_domain, $image["responsive_url"], 2)[1];
if(substr($thumbnail, 0, 3) != "/1/"){
$thumbnail =
preg_replace(
'/^\/[^\/]+/',
"",
$thumbnail
);
}
$thumbnail = "https://img.vsco.co/cdn-cgi/image/width=480,height=360" . $thumbnail;
$size =
$this->image_ratio(
(int)$image["dimensions"]["width"],
(int)$image["dimensions"]["height"]
);
$out["image"][] = [
"title" => $image["description"],
"source" => [
[
"url" => "https://" . $image["responsive_url"],
"width" => (int)$image["dimensions"]["width"],
"height" => (int)$image["dimensions"]["height"]
],
[
"url" => $thumbnail,
"width" => $size[0],
"height" => $size[1]
]
],
"url" => "https://" . $image["grid"]["domain"] . "/media/" . $image["imageId"]
];
}
// get NPT
$max_page = ceil($json["total"] / 100);
$data["pagination"]["page"]++;
if($max_page > $data["pagination"]["page"]){
$out["npt"] =
$this->backend->store(
json_encode($data),
"images",
$proxy
);
}
return $out;
}
private function image_ratio($width, $height){
$ratio = [
480 / $width,
360 / $height
];
if($ratio[0] < $ratio[1]){
$ratio = $ratio[0];
}else{
$ratio = $ratio[1];
}
return [
floor($width * $ratio),
floor($height * $ratio)
];
}
}

View File

@ -37,7 +37,7 @@ class yandex{
"Accept-Encoding: gzip",
"Accept-Language: en-US,en;q=0.5",
"DNT: 1",
"Cookie: yp=1716337604.sp.family%3A{$nsfw}#1685406411.szm.1:1920x1080:1920x999",
"Cookie: yp=" . (time() - 4000033) . ".szm.1:1920x1080:876x1000#" . time() . ".sp.family:" . $nsfw,
"Referer: https://yandex.com/images/search",
"Connection: keep-alive",
"Upgrade-Insecure-Requests: 1",
@ -668,7 +668,6 @@ class yandex{
foreach($json["blocks"] as $block){
$html .= $block["html"];
// get next page
if(
isset($block["params"]["nextPageUrl"]) &&

View File

@ -1209,15 +1209,16 @@ class yt{
$reel =
$reel
->reelItemRenderer;
->shortsLockupViewModel;
array_push(
$this->out["reel"],
[
"title" =>
$reel
->headline
->simpleText,
->overlayMetadata
->primaryText
->content,
"description" => null,
"author" => [
"name" => null,
@ -1225,30 +1226,22 @@ class yt{
"avatar" => null
],
"date" => null,
"duration" =>
$this->textualtime2int(
$reel
->accessibility
->accessibilityData
->label
),
"views" =>
$this->truncatedcount2int(
$reel
->viewCountText
->simpleText
),
"duration" => null,
"views" => null,
"thumb" => [
"url" =>
$reel
->thumbnail
->thumbnails[0]
->sources[0]
->url,
"ratio" => "9:16"
],
"url" =>
"https://www.youtube.com/watch?v=" .
$reel
->onTap
->innertubeCommand
->reelWatchEndpoint
->videoId
]
);

View File

@ -133,6 +133,10 @@ $settings = [
"value" => "google",
"text" => "Google"
],
[
"value" => "google_cse",
"text" => "Google CSE"
],
[
"value" => "startpage",
"text" => "Startpage"
@ -203,6 +207,10 @@ $settings = [
"value" => "google",
"text" => "Google"
],
[
"value" => "google_cse",
"text" => "Google CSE"
],
[
"value" => "startpage",
"text" => "Startpage"
@ -219,10 +227,22 @@ $settings = [
"value" => "solofield",
"text" => "Solofield"
],
/*[
[
"value" => "pinterest",
"text" => "Pinterest"
],*/
],
[
"value" => "flickr",
"text" => "Flickr"
],
[
"value" => "fivehpx",
"text" => "500px"
],
[
"value" => "vsco",
"text" => "VSCO"
],
[
"value" => "imgur",
"text" => "Imgur"

View File

@ -1,31 +1,30 @@
:root{
--1d2021: #1d2021;
--282828: #282828;
--3c3836: #3c3836;
--504945: #504945;
--1d2021:#1d2021;
--282828:#282828;
--3c3836:#3c3836;
--504945:#504945;
/* font */
--928374: #928374;
--a89984: #c9c5bf;
--bdae93: #bdae93;
--8ec07c: #8ec07c;
--ebdbb2: #ebdbb2;
--928374:#928374;
--a89984:#c9c5bf;
--bdae93:#bdae93;
--8ec07c:#8ec07c;
--ebdbb2:#ebdbb2;
}
body{
padding:15px 4% 40px;
margin:unset;
}
h1,h2,h3,h4,h5,h6{
h1, h2, h3, h4, h5, h6{
padding:0;
margin:0 0 7px 0;
line-height:initial;
color:var(--bdae93);
}
h3,h4,h5,h6{
h3, h4, h5, h6{
margin-bottom:14px;
}
@ -37,11 +36,10 @@ h3,h4,h5,h6{
float:right;
cursor:pointer;
padding:0 10px;
border-left: 1px solid var(--504945);
background: #723c0b;
border-left:1px solid var(--504945);
background:#723c0b;
}
.searchbox input{
all:unset;
line-height:36px;
@ -96,7 +94,6 @@ h3,h4,h5,h6{
display:inline-block;
}
.tabs .tab.selected{
border-bottom:2px solid #fc92a5;
}
@ -106,7 +103,7 @@ h3,h4,h5,h6{
padding-bottom:12px;
padding-top:7px;
margin-bottom:7px;
background-color:#232525
background-color:#232525;
}
.filters .filter{
@ -169,7 +166,6 @@ h3,h4,h5,h6{
font-size:12px;
}
.web .hover{
display:block;
text-decoration:none;
@ -193,16 +189,13 @@ h3,h4,h5,h6{
color:#9760b1 !important;
}
.web .text-result .greentext{
font-size:14px;
color:var(--bdae93);
}
/* favicon */
.favicon-dropdown a{
text-decoration:none;
color:#d3d0c1;
@ -211,37 +204,31 @@ h3,h4,h5,h6{
font-size:13px;
}
.web .favicon img,
.favicon-dropdown img{
.web .favicon img, .favicon-dropdown img{
margin:3px 7px 0 0;
height:16px;
font-size:12px;
line-height:16px;;
line-height:16px;
display:block;
text-align:left;
}
.web .sublinks{
padding:17px 10px;
font-size:15px;
color:var(--#928374);
}
.web .text-result .sublinks:last-child{
padding-bottom:0;
}
/* Wikipedia head */
.wiki-head{
padding:5px;
background-color: #322f2b
background-color:#322f2b;
}
/*
Images tab
*/
@ -257,8 +244,6 @@ h3,h4,h5,h6{
float:left;
}
#images .image .title{
white-space:nowrap;
overflow:hidden;
@ -267,7 +252,6 @@ h3,h4,h5,h6{
color:var(--bdae93);
}
#popup-status{
display:none;
position:fixed;
@ -283,40 +267,56 @@ h3,h4,h5,h6{
Settings page
*/
.web .settings-submit a{
margin-right:17px;
color:#bdae93;
}
/*
Responsive image
*/
@media only screen and (max-width: 1454px){ #images .image-wrapper{ width:25%; } }
@media only screen and (max-width: 1161px){ #images .image-wrapper{ width:25%; } }
@media only screen and (max-width: 750px){ #images .image-wrapper{ width:50%; } }
@media only screen and (max-width: 450px){ #images .image-wrapper{ width:100%; } }
@media only screen and (max-width:1454px){
#images .image-wrapper{
width:25%;
}
}
@media only screen and (max-width:1161px){
#images .image-wrapper{
width:25%;
}
}
@media only screen and (max-width:750px){
#images .image-wrapper{
width:50%;
}
}
@media only screen and (max-width:450px){
#images .image-wrapper{
width:100%;
}
}
/*
Responsive design
*/
@media only screen and (max-width: 1550px){
.web .left,
@media only screen and (max-width:1550px){
.web .left,
.searchbox{
width:60%;
}
}
@media only screen and (max-width: 1000px){
}
.type{
color:var(--bdae93);
@media only screen and (max-width:1100px){
.web .left,
.searchbox{
width:100%;
}
}
.type{
color:var(--bdae93);
}

40
static/themes/Wine.css Normal file
View File

@ -0,0 +1,40 @@
:root
{
--accent : #f79e98;
--1d2021 : #180d0c;
--282828 : #180d0c;
--3c3836 : #251615;
--504945 : #251615;
--928374 : var(--accent);
--a89984 : #d8c5c4;
--bdae93 : #d8c5c4;
--8ec07c : var(--accent);
--ebdbb2 : #d8c5c4;
--comment: #928374;
--default: #DCC9BC;
--keyword: #F07342;
--string : var(--accent);
--green : #959A6B;
--yellow : #E39C45;
--red : #CF223E;
--white : var(--a89984);
--black : var(--1d2021);
--hover : #b18884
}
a.link, a { color: var(--accent); text-decoration: none; }
.searchbox { width: 23%; }
.filters filter select { color: #E39C45; }
.web .separator::before { color: var(--white) }
.searchbox input[type="text"]::placeholder { color: var(--white); }
a.link:hover
{
color: var(--hover);
text-shadow: 0 0 .2rem var(--hover);
}
.code-inline
{ border-color: var(--default); font-family: monospace;}
.home #center a
{ color: var(--accent); }
.home .subtext
{ color: var(--white); }