lolcat
/
4get
5
13
Fork 4
Code Issues 11 Pull Requests Packages Projects Releases Wiki Activity
4get/README.md

11 KiB
Raw Blame History

ko-fi

4get

4get is a metasearch engine that doesn't suck (they live in our walls!)

About 4get

https://4get.ca/about

Try it out

https://4get.ca

Totally unbiased comparison between alternatives

4get searx(ng) librex araa
RAM usage 200-400mb~ 2GB~ 200-400mb~ 2GB~
Does it suck no (debunked by snopes) yes yes a little
Does it work ye no no ye
Did the dev commit suicide not until my 30s idk yes no

Supported websites

  1. Web

    • DuckDuckGo
    • Brave
    • Yandex
    • Google
    • Mojeek
    • Marginalia
    • wiby

  2. Images

    • DuckDuckGo
    • Yandex
    • Google
    • Brave
    • Yep
    • Imgur
    • FindThatMeme

  3. Videos

    • YouTube
    • DuckDuckgo
    • Brave
    • Yandex
    • Google

  4. News

    • DuckDuckGo
    • Brave
    • Google
    • Mojeek

  5. Music

    • SoundCloud

  6. Autocompleter

    • Brave
    • DuckDuckGo
    • Yandex
    • Google
    • Qwant
    • Yep
    • Marginalia
    • YouTube
    • SoundCloud

More scrapers are coming soon. I currently want to add HackerNews (durr orange site!!), Qwant, Yep and other garbage. A shopping, files, tab and more music scrapers are also on my todo list.

Installation

This section is still to-do. You will need to figure shit out for some of the apache2 and nginx stuff. Everything else should be OK.

Install on Apache

Login as root.

apt install php-mbstring apache2 certbot php-imagick imagemagick php-curl curl php-apcu git libapache2-mod-php python3-certbot-apache
service apache2 start
a2enmod rewrite

For all of the files in /etc/apache2/sites-enabled/, you must apply the following changes:

  • Uncomment ServerName directive, put your domain name there
  • Change ServerAdmin to your email
  • Change DocumentRoot to /var/www/html/4get
  • Change ErrorLog and CustomLog directives to log stuff out to /dev/null/

Now open /etc/apache2/apache2.conf and change ErrorLog and CustomLog directives to have /dev/null/ as a value

This should disable logging completely, but I'm not 100% sure since I sort of had to troubleshoot alot of shit while writing this. So after we're done check if /var/log/apache2/* contains any personal info, and if it does, call me retarded trough email exchange.

Blindly run the following shit

cd /var/www/html
git clone https://git.lolcat.ca/lolcat/4get
cd 4get
mkdir icons
chmod 777 -R icons/

Restart the service for good measure... service apache2 restart

Install on NGINX

Login as root.

Create a file in /etc/nginx/sites-avaliable/ called 4get.conf or any name you want and put this into the file:

server {
    # DO YOU REALLY NEED TO LOG SEARCHES?
    access_log /dev/null;
    error_log /dev/null;
    # Change this if you have 4get in other folder.
    root /var/www/4get;
    # Change yourdomain by your domain lol
    server_name www.yourdomain.com yourdomain.com;

    location @php {
        try_files $uri.php $uri/index.php =404;
                # Change the unix socket address if it's different for you.
        fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
        fastcgi_index index.php;
                # Change this to `fastcgi_params` if you use a debian based distro.
        include fastcgi.conf;
        fastcgi_intercept_errors on;
    }

    location / {
        try_files $uri @php;
    }

    location ~* ^(.*)\.php$ {
        return 301 $1;
    }

        listen 80;
}

That is a very basic config so you will need to adapt it to your needs in case you have a more complicated nginx configuration. Anyways, you can see a real world example here

After you save the file you will need to do a symlink of the 4get.conf file to /etc/nignx/sites-enabled/, you can do it with this command:

ln -s /etc/nginx/sites-available/4get.conf /etc/nginx/sites-available/4get.conf

Now test the nginx config with nginx -t, if it says that everything is good, restart nginx using systemctl restart nginx

Install using Docker (lol u lazy fuck)

docker run -d -p 80:80 -e FOURGET_SERVER_NAME="4get.ca" luuul/4get:latest

...Or with SSL:

docker run -d -p 443:443 -v /etc/letsencrypt/live/domain.tld:/etc/4get/certs -e FOURGET_SERVER_NAME="4get.ca" luuul/4get:latest

if the certificate files are not mounted to /etc/4get/certs the service listens to port 80

the certificate directory expects files named fullchain.pem and privkey.pem

Install using Docker Compose

copy docker-compose.yaml

to serve custom banners create a directory named banners for example with images and mount to /var/www/html/4get/banner

to serve captcha images create a directory named captchas for example containing subfolders with images and mount to /var/www/html/4get/data/captcha

any environment variables prefixed with FOURGET_ will be added to the generated config

the entrypoint will automatically set the CAPTCHA_DATASET value for you based on directory names and number of files in each

to set INSTANCES pass a comma separated string of urls (FOURGET_INSTANCES = "https://4get.ca,https://domain.tld")

version: "3.7"

services:
  fourget:
    image: luuul/4get:latest
    restart: always
    environment:
      - FOURGET_SERVER_NAME=4get.ca

    ports:
      - "80:80"
      - "443:443"

    volumes:
      - /etc/letsencrypt/live/domain.tld:/etc/4get/certs
      - ./banners:/var/www/html/4get/banner
      - ./captchas:/var/www/html/4get/data/captcha

Replace relevant values and start with docker compose up -d

Install on Caddy

  1. Install dependencies:

sudo apt install caddy php8.2-dom php8.2-imagick imagemagick php8.2-curl curl php8.2-apcu git

  1. Clone this repository where you want to host this from:

cd /var/www && sudo git clone https://git.konakona.moe/diowo/4get

  1. Set permission on the icons directory inside 4get

cd /var/www/4get/ && sudo chmod 777 -R icons/

  1. Add an entry for 4get on your Caddyfile at /etc/caddy/Caddyfile
4get.konakona.moe {
    root * /var/www/4get
    file_server
    encode gzip
    php_fastcgi unix//var/run/php/php8.2-fpm.sock {
        index index.php
    }
    redir /{path}.php{query} 301
    try_files {path} {path}.php
}

Caddy deals with SSL certificates automatically so you don't have to mess with anything. Also if needed, a sample of my Caddyfile can be found here.

  1. Restart Caddy

sudo systemctl restart caddy

Encryption setup

I'm schizoid (as you should) so I'm gonna setup 4096bit key encryption. To complete this step, you need a domain or subdomain in your possession. Make sure that the DNS shit for your domain has propagated properly before continuing, because certbot is a piece of shit that will error out the ass once you reach 5 attempts under an hour.

Encryption setup on Apache

certbot --apache --rsa-key-size 4096 -d www.yourdomain.com -d yourdomain.com

When it asks to choose a vhost, choose the option with "HTTPS" listed. Don't setup HTTPS for tor, we don't need it (it doesn't even work anyways with let's encrypt)

Edit 000-default-le-ssl.conf

Add this at the end:

<Directory /var/www/html/4get>
	RewriteEngine On
	RewriteCond %{REQUEST_FILENAME}.php -f
	RewriteRule (.*) $1.php [L]
	Options Indexes FollowSymLinks
	AllowOverride All
	Require all granted
</Directory>

Now since this file is located in /etc/apache2/sites-enabled/, you must change all of the logging shit as to make it not log anything, like we did earlier.

Restart again

service apache2 restart

Encryption setup on NGINX

Generate a certificate for the domain using:

certbot --nginx --key-type ecdsa -d www.yourdomain.com -d yourdomain.com

(Remember to install the nginx certbot plugin!!!)

After doing that certbot should deploy the certificate automatically into your 4get nginx config file. It should be ready to use at that point.

Jesse it is time to configure the server the fucking bots are back

Wohoo the awful piece of shit setup and fiddling with 3 gazillion files is GONE. All you need to do to configure your shit is to go in data/config.php and edit the self-documenting configuration file. You can also specify proxies in data/proxies/whatever.txt and captcha images in data/captcha/category/1.png... I further explain how to deal with that garbage in the config file I mentionned.

(Optional) Tor setup

  1. Install tor.
  2. Open /etc/tor/torrc
  3. Go to the line that contains HiddenServiceDir and HiddenServicePort
  4. Uncomment those 2 lines and set them like this: 
    HiddenServiceDir /var/lib/tor/4get
HiddenServicePort 80 127.0.0.1:80
  5. Start the tor service using systemctl start tor
  6. Wait some seconds...
  7. Login as root and execute this command: cat /var/lib/tor/4get/hostname
  8. That is your onion address.

After you get your onion address you will need to configure your Apache or Nginx config or you will get 404 errors.

I don't know to configure this shit on Apache so here is the NGINX one.

Tor setup on NGINX

Important Note: Tor onion addresses are significantly longer than traditional domain names. Before proceeding with Nginx configuration, ensure you increase the server_names_hash_bucket_size value in your nginx.conf file. This setting in your Nginx configuration controls the internal data structure used to manage multiple server names (hostnames) associated with your web server. Each hostname requires a certain amount of memory within this structure. If the size is insufficient, Nginx will encounter errors.

  1. Open your nginx.conf file (that is under /etc/nginx/nginx.conf).
  2. Find the line containing # server_names_hash_bucket_size 64;.
  3. Uncomment the line and adjust the value. Start with 64, but if you encounter issues, incrementally increase it (e.g., 128, 256) until it accommodates your configuration.

Open your current 4get NGINX config (that is under /etc/nginx/sites-available/) and append this to the end of the file:

server {
	access_log /dev/null;
	error_log /dev/null;

    listen 80;
    server_name <youronionaddress>;
    root /var/www/4get;

    location @php {
        try_files $uri.php $uri/index.php =404;
        # Change the unix socket address if it's different for you.
        fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
        fastcgi_index index.php;
        # Change this to `fastcgi_params` if you use a debian based distro.
        include fastcgi.conf;
        fastcgi_intercept_errors on;
    }

    location / {
        try_files $uri @php;
    }

    location ~* ^(.*)\.php$ {
        return 301 $1;
    }
}

Obviously replace <youronionaddress> by the onion address of /var/lib/tor/4get/hostname and then check if the nginx config is valid with nginx -t if yes, then restart the nginx service and try opening the onion address into the Tor Browser. You can see a real world example here

Contact

shit breaks all the time but I repair it all the time too. Email me here: willlolcat(dot)ca