Question: how set 4get instance in Artix #19

Closed
opened 2024-06-09 17:52:39 +00:00 by root · 70 comments

I am looking the guide https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/apache2.md.
Artix instructions should be the same for Arch (except the service commands).

I did the step before https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/apache2.md#default-sslconf but I may have missed something.

/etc/httpd/confextra/httpd-ssl.conf:

<VirtualHost *:443>
	RedirectMatch 301 ^(.*)$ https://http://localhost/c.ca/$1
	ServerAdmin user@c.ca

	#ErrorLog ${APACHE_LOG_DIR}/error.log

	SSLEngine on

	<FilesMatch "\.(?:cgi|shtml|phtml|php)$">
		SSLOptions +StdEnvVars
	</FilesMatch>
	<Directory /usr/lib/cgi-bin>
		SSLOptions +StdEnvVars
	</Directory>

	#AddOutputFilterByType DEFLATE application/json
	#AddOutputFilterByType DEFLATE application/javascript
	#AddOutputFilterByType DEFLATE application/x-javascript
	#AddOutputFilterByType DEFLATE text/html
	#AddOutputFilterByType DEFLATE text/plain
	#AddOutputFilterByType DEFLATE text/css

	#SSLCertificateFile /etc/letsencrypt/live/4get.ca/fullchain.pem
	#SSLCertificateKeyFile /etc/letsencrypt/live/4get.ca/privkey.pem
</VirtualHost>

<VirtualHost *:443>
	ServerName c.ca

	DocumentRoot /srv/http/c

	Options +MultiViews
	RewriteEngine On
	RewriteCond %{REQUEST_FILENAME} !-d
	RewriteCond %{REQUEST_FILENAME} !-f
	RewriteRule ^([^\.]+)$ $1.php [NC,L]

	# deny access to private resources
	<Directory /srv/http/c/>
		Order Deny,allow
		Deny from all
	</Directory>
</VirtualHost>

<VirtualHost *:443>
	ServerName c.ca

	DocumentRoot /srv/http/c

	Options +MultiViews
	RewriteEngine On
	RewriteCond %{REQUEST_FILENAME} !-d
	RewriteCond %{REQUEST_FILENAME} !-f
	RewriteRule ^([^\.]+)$ $1.php [NC,L]
</VirtualHost>

<VirtualHost *:443>
	ServerName www.c.ca
	RedirectMatch 301 ^(.*)$ https://c.ca$1
</VirtualHost>

I enabled the file commenting out Include conf/extra/httpd-ssl.conf in /etc/httpd/conf/httpd.conf

I had errors about DEFLATE so I commented it.

AH00526: Syntax error on line 16 of /etc/httpd/conf/extra/httpd-ssl.conf:
Unknown filter provider DEFLATE

Then I commented also SSLCertificateFile and SSLCertificateFile because I do not have them yet.
Apache won't start and I do not see anymore errors in the log.

This is how I cloned the repo.

cd /srv/http
git clone https://git.lolcat.ca/lolcat/4get
mv 4get c
cd c
chmod 777 -R icons/

I am not sure if I specified DocumentRoot /srv/http/c, RedirectMatch 301 ^(.*)$ https://c.ca$1 etc. correctly.

Is it fine asking here?

I am looking the guide https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/apache2.md. Artix instructions should be the same for Arch (except the service commands). I did the step before https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/apache2.md#default-sslconf but I may have missed something. `/etc/httpd/confextra/httpd-ssl.conf`: ``` <VirtualHost *:443> RedirectMatch 301 ^(.*)$ https://http://localhost/c.ca/$1 ServerAdmin user@c.ca #ErrorLog ${APACHE_LOG_DIR}/error.log SSLEngine on <FilesMatch "\.(?:cgi|shtml|phtml|php)$"> SSLOptions +StdEnvVars </FilesMatch> <Directory /usr/lib/cgi-bin> SSLOptions +StdEnvVars </Directory> #AddOutputFilterByType DEFLATE application/json #AddOutputFilterByType DEFLATE application/javascript #AddOutputFilterByType DEFLATE application/x-javascript #AddOutputFilterByType DEFLATE text/html #AddOutputFilterByType DEFLATE text/plain #AddOutputFilterByType DEFLATE text/css #SSLCertificateFile /etc/letsencrypt/live/4get.ca/fullchain.pem #SSLCertificateKeyFile /etc/letsencrypt/live/4get.ca/privkey.pem </VirtualHost> <VirtualHost *:443> ServerName c.ca DocumentRoot /srv/http/c Options +MultiViews RewriteEngine On RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !-f RewriteRule ^([^\.]+)$ $1.php [NC,L] # deny access to private resources <Directory /srv/http/c/> Order Deny,allow Deny from all </Directory> </VirtualHost> <VirtualHost *:443> ServerName c.ca DocumentRoot /srv/http/c Options +MultiViews RewriteEngine On RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !-f RewriteRule ^([^\.]+)$ $1.php [NC,L] </VirtualHost> <VirtualHost *:443> ServerName www.c.ca RedirectMatch 301 ^(.*)$ https://c.ca$1 </VirtualHost> ``` I enabled the file commenting out `Include conf/extra/httpd-ssl.conf` in `/etc/httpd/conf/httpd.conf` I had errors about `DEFLATE` so I commented it. ``` AH00526: Syntax error on line 16 of /etc/httpd/conf/extra/httpd-ssl.conf: Unknown filter provider DEFLATE ``` Then I commented also `SSLCertificateFile` and `SSLCertificateFile` because I do not have them yet. Apache won't start and I do not see anymore errors in the log. This is how I cloned the repo. ``` cd /srv/http git clone https://git.lolcat.ca/lolcat/4get mv 4get c cd c chmod 777 -R icons/ ``` I am not sure if I specified `DocumentRoot /srv/http/c`, `RedirectMatch 301 ^(.*)$ https://c.ca$1` etc. correctly. Is it fine asking here?
Owner

Asking here is fine, lol

For the deflate error, please try enabling the extension using sudo a2enmod enable deflate or something like that. Also try with mod-deflate if that doesn't work. If the module doesn't exist, try and look in your package manager for apache2 package extras.

That RedirectMatch directive will redirect all traffic on port 443 (https, assuming SSLEngine is on) to whatever domain you specify. I'm not sure what you're trying to do here, are you trying to only allow traffic through localhost? If so, leave out the ssl config stuff. If not, try and fix it like this, assuming you're using the c.ca domain:

RedirectMatch 301 ^(.*)$ https://c.ca/$1

Again, this will redirect all SSL traffic to your c.ca domain, so make sure that this is what you want.

Now, Apache2 won't start because there is no SSLCertificateFile while SSLEngine is on. Generate your certificates and uncomment the SSLCertificateFile directives, all the while making sure you supply the right file paths.

Also, not quite sure why you have 2 rulesets with ServerName c.ca? It's better to merge them to avoid conflicts.

Keep me posted!

Asking here is fine, lol For the deflate error, please try enabling the extension using `sudo a2enmod enable deflate` or something like that. Also try with `mod-deflate` if that doesn't work. If the module doesn't exist, try and look in your package manager for apache2 package extras. That `RedirectMatch` directive will redirect all traffic on port 443 (https, assuming `SSLEngine` is on) to whatever domain you specify. I'm not sure what you're trying to do here, are you trying to only allow traffic through `localhost`? If so, leave out the ssl config stuff. If not, try and fix it like this, assuming you're using the `c.ca` domain: `RedirectMatch 301 ^(.*)$ https://c.ca/$1` Again, this will redirect all SSL traffic to your c.ca domain, so make sure that this is what you want. Now, Apache2 won't start because there is no `SSLCertificateFile` while `SSLEngine` is on. Generate your certificates and uncomment the `SSLCertificateFile` directives, all the while making sure you supply the right file paths. Also, not quite sure why you have 2 rulesets with `ServerName c.ca`? It's better to merge them to avoid conflicts. Keep me posted!
Author

I'd like to set up my 4get instance that I can use locally so I supposed that you use https://http://localhost/c.ca in these cases.
I am not sure to how set up the local instance.

I made 2 rulesets trying to following the guide but I am not familiar with this.

In /etc/httpd/conf/httpd.conf I now commented out LoadModule deflate_module modules/mod_deflate.so, looks like that a2enmod is Debian specific.

In this case do I need ssl config?

I'd like to set up my 4get instance that I can use locally so I supposed that you use `https://http://localhost/c.ca` in these cases. I am not sure to how set up the local instance. I made 2 rulesets trying to following the guide but I am not familiar with this. In `/etc/httpd/conf/httpd.conf` I now commented out `LoadModule deflate_module modules/mod_deflate.so`, looks like that `a2enmod` is Debian specific. In this case do I need ssl config?
Owner

You do not need an ssl config for local use. Why do you insist on using https://http://localhost/c.ca ? This is wrong, you specify the protocol 2 times. Since you're running this locally, there is no need to setup redirects, you can just access your webserver directly with http://localhost .

Please let me know if you have any other questions :)

You do not need an ssl config for local use. Why do you insist on using `https://http://localhost/c.ca` ? This is wrong, you specify the protocol 2 times. Since you're running this locally, there is no need to setup redirects, you can just access your webserver directly with http://localhost . Please let me know if you have any other questions :)
Author

Do I need to use /etc/httpd/conf/extra/httpd-vhosts.conf?

How do I make 4get instance visible on localhost?

Do I need to use `/etc/httpd/conf/extra/httpd-vhosts.conf`? How do I make 4get instance visible on localhost?
Owner

It should be visible by default. The only thing you need to do is the url's .php extension internal redirect (the RewriteRule directives)

It should be visible by default. The only thing you need to do is the url's .php extension internal redirect (the `RewriteRule` directives)
Author

Could you make an example of all the things that I need?
Not sure where put those rules in /etc/httpd/conf/httpd.conf?

Or

ls /etc/httpd/conf/extra/
httpd-autoindex.conf  httpd-info.conf       httpd-mpm.conf                 httpd-userdir.conf
httpd-dav.conf        httpd-languages.conf  httpd-multilang-errordoc.conf  httpd-vhosts.conf
httpd-default.conf    httpd-manual.conf     httpd-ssl.conf                 proxy-html.conf
Could you make an example of all the things that I need? Not sure where put those rules in `/etc/httpd/conf/httpd.conf`? Or ``` ls /etc/httpd/conf/extra/ httpd-autoindex.conf httpd-info.conf httpd-mpm.conf httpd-userdir.conf httpd-dav.conf httpd-languages.conf httpd-multilang-errordoc.conf httpd-vhosts.conf httpd-default.conf httpd-manual.conf httpd-ssl.conf proxy-html.conf ```
Owner

At this point, what I would suggest doing would be to run dpkg-reconfigure on your apache2 webserver and create an .htaccess configuration file in your /var/www/4get/ folder that can get rid of the .php extension by doing an internal URL rewrite. This is all you need to do in order to make 4get work locally.

You can look up how to enable .htaccess online and then just place the following rules in there:

Options +MultiViews
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^\.]+)$ $1.php [NC,L]

When you make configuration changes, make sure to restart apache2 every time.

Please let me know if you encounter any other issues.

At this point, what I would suggest doing would be to run `dpkg-reconfigure` on your apache2 webserver and create an `.htaccess` configuration file in your `/var/www/4get/` folder that can get rid of the `.php` extension by doing an internal URL rewrite. This is all you need to do in order to make 4get work locally. You can look up how to enable .htaccess online and then just place the following rules in there: ``` Options +MultiViews RewriteEngine On RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !-f RewriteRule ^([^\.]+)$ $1.php [NC,L] ``` When you make configuration changes, make sure to restart apache2 every time. Please let me know if you encounter any other issues.
Author

https://httpd.apache.org/docs/current/howto/htaccess.html#when

I made this in the conf and I also made the equivalent .htaccess file in /srv/http/4get.

<Directory "/srv/http/4get">
    Options +MultiViews
    RewriteEngine On
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule ^([^\.]+)$ $1.php [NC,L]
</Directory>

/srv/http is the defautl value in the conf, and I enabled AllowOverride:

DocumentRoot "/srv/http"
<Directory "/srv/http">
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
</Directory>

<Directory />
    AllowOverride All
    Require all denied
</Directory>

When I go to localhost I see the 4get folder and I can browse it but I do not know to how use it as search engine.

For local use do I need all the packages php-mbstring apache2 certbot php-imagick imagemagick php-curl curl php-apcu git libapache2-mod-php?
And ssl, rewrite, proxy and headers modules?

https://httpd.apache.org/docs/current/howto/htaccess.html#when I made this in the conf and I also made the equivalent `.htaccess` file in `/srv/http/4get`. ``` <Directory "/srv/http/4get"> Options +MultiViews RewriteEngine On RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !-f RewriteRule ^([^\.]+)$ $1.php [NC,L] </Directory> ``` `/srv/http` is the defautl value in the conf, and I enabled `AllowOverride`: ``` DocumentRoot "/srv/http" <Directory "/srv/http"> Options Indexes FollowSymLinks AllowOverride All Require all granted </Directory> <Directory /> AllowOverride All Require all denied </Directory> ``` When I go to localhost I see the 4get folder and I can browse it but I do not know to how use it as search engine. For local use do I need all the packages `php-mbstring apache2 certbot php-imagick imagemagick php-curl curl php-apcu git libapache2-mod-php`? And ssl, rewrite, proxy and headers modules?
Owner

You need php-mbstring apache2 php-imagick imagemagick php-curl curl php-apcu libapache2-mod-php and rewrite modules to get rid of the .php extension.

You need `php-mbstring apache2 php-imagick imagemagick php-curl curl php-apcu libapache2-mod-php` and rewrite modules to get rid of the .php extension.
Author

Right now I do not have php-mbstring and php-curl.

Is it supposed to show up at least?

Right now I do not have `php-mbstring` and `php-curl`. Is it supposed to show up at least?
Owner

I don't really understand why you're having this many issues. I have followed these installation steps myself multiple times now while migrating machines and I can assure you that they work.

When you access your webserver, does anything render, or is it just a directory listing? When you access /index.php, what happens?

I don't really understand why you're having this many issues. I have followed these installation steps myself multiple times now while migrating machines and I can assure you that they work. When you access your webserver, does anything render, or is it just a directory listing? When you access `/index.php`, what happens?
Author

It is just a directory listing.

When I access http://127.0.1.1/4get/index.php I see an empty page with this line at the beginning:

load( "home.html", [ "server_short_description" => htmlspecialchars(config::SERVER_SHORT_DESCRIPTION), "banner" => $images[rand(0, count($images) - 1)] ] );

It is just a directory listing. When I access `http://127.0.1.1/4get/index.php` I see an empty page with this line at the beginning: > load( "home.html", [ "server_short_description" => htmlspecialchars(config::SERVER_SHORT_DESCRIPTION), "banner" => $images[rand(0, count($images) - 1)] ] );
Owner

Did you install libapache2-mod-php?

Did you install `libapache2-mod-php`?
Author

I installed this that I believe being the corresponding version:

pacman -Ql php-apache 
php-apache /etc/
php-apache /etc/httpd/
php-apache /etc/httpd/conf/
php-apache /etc/httpd/conf/extra/
php-apache /etc/httpd/conf/extra/php_module.conf
php-apache /usr/
php-apache /usr/lib/
php-apache /usr/lib/httpd/
php-apache /usr/lib/httpd/modules/
php-apache /usr/lib/httpd/modules/libphp.so

I now added to the conf:

Include conf/extra/php_module.conf
LoadModule php_module /usr/lib/httpd/modules/libphp.so

It does not start anymore:
Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe. You need to recompile PHP.

Do I need to recompile it or should I do something else?

Disabling mpm_event and enabling mpm_prefork makes apache starts but I see what I was seeing before.

#LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
I installed this that I believe being the corresponding version: ``` pacman -Ql php-apache php-apache /etc/ php-apache /etc/httpd/ php-apache /etc/httpd/conf/ php-apache /etc/httpd/conf/extra/ php-apache /etc/httpd/conf/extra/php_module.conf php-apache /usr/ php-apache /usr/lib/ php-apache /usr/lib/httpd/ php-apache /usr/lib/httpd/modules/ php-apache /usr/lib/httpd/modules/libphp.so ``` I now added to the conf: ``` Include conf/extra/php_module.conf LoadModule php_module /usr/lib/httpd/modules/libphp.so ``` It does not start anymore: ` Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe. You need to recompile PHP.` Do I need to recompile it or should I do something else? Disabling mpm_event and enabling mpm_prefork makes apache starts but I see what I was seeing before. ``` #LoadModule mpm_event_module modules/mod_mpm_event.so LoadModule mpm_prefork_module modules/mod_mpm_prefork.so ```
Owner

What the fuck kind of distro ships broken apache2 versions that can't run PHP? Don't recompile PHP, that's a whole new rabbit hole that I swear you do not want to get involved in. This shit is asking you to enable ZTS support, which is an absolute fucking pain in the ass to setup.

Your only option would be to try installing 4get through docker at this point. This is insane.

What the fuck kind of distro ships broken apache2 versions that can't run PHP? Don't recompile PHP, that's a whole new rabbit hole that I *swear* you do not want to get involved in. This shit is asking you to enable ZTS support, which is an absolute fucking pain in the ass to setup. Your only option would be to try installing 4get through docker at this point. This is insane.
Author

Does it need mpm_event therefore?

Since disabling mpm_event and enabling mpm_prefork does not have errors.

I'll search to how do it for Artix/Arch otherwise.

Does it need mpm_event therefore? Since disabling mpm_event and enabling mpm_prefork does not have errors. I'll search to how do it for Artix/Arch otherwise.
Owner

Dude I'm so sorry you're going through this, I have no fucking clue what mpm_event does. Shit just works on debian, and docker works on pretty much anything.

Let me know if you figure this out, I might write a guide for Artix users or something..

Dude I'm so sorry you're going through this, I have no fucking clue what `mpm_event` does. Shit just works on debian, and docker works on pretty much anything. Let me know if you figure this out, I might write a guide for Artix users or something..
Author

I did not look the Arch wiki after adding AddHandler php-script .php in the conf it starts.

I installed php83-curl and php83-mbstring from the AUR, those needed php83-8.3.7 too.
Not sure how that is supposed to work when php in repo is just php.

Now it shows up but it does not work, I attach screenshots.

I did not look the [Arch wiki]( https://wiki.archlinux.org/title/Apache_HTTP_Server#Using_libphp) after adding `AddHandler php-script .php` in the conf it starts. I installed `php83-curl` and `php83-mbstring` from the AUR, those needed `php83-8.3.7` too. Not sure how that is supposed to work when php in repo is just php. Now it shows up but it does not work, I attach screenshots.
Owner

4get should be stored at the root of the webserver, otherwise you will see broken links.

/settings returning a 404 error means the .php internal redirect does not work.

4get should be stored at the root of the webserver, otherwise you will see broken links. /settings returning a 404 error means the .php internal redirect does not work.
Author

I moved the files from /srv/http/4get to /srv/http/.

And I updated the conf:

ServerRoot "/etc/httpd"
...
<Directory "/srv/http/">
...

The home page is displayed better but settings, searching something and others still gets 404 error as before.

I moved the files from `/srv/http/4get` to `/srv/http/`. And I updated the conf: ``` ServerRoot "/etc/httpd" ... <Directory "/srv/http/"> ... ``` The home page is displayed better but settings, searching something and others still gets 404 error as before.
Owner

Setup the RedirectRule config

Setup the RedirectRule config
Author

Not sure to how do that.

Is it similar to this?

<VirtualHost *:80>
	# no domain = go to 4get.ca
	RedirectMatch 301 ^(.*)$ https://4get.ca$1
</VirtualHost>
Not sure to how do that. Is it similar to this? ``` <VirtualHost *:80> # no domain = go to 4get.ca RedirectMatch 301 ^(.*)$ https://4get.ca$1 </VirtualHost> ```
Owner

No, you use the rules here

#19 (comment)

No, you use the rules here https://git.lolcat.ca/lolcat/4get/issues/19#issuecomment-75
Author

I am using this in the main config:

DocumentRoot "/srv/http/4get"

<Directory "/srv/http/4get">
    Options +MultiViews
    RewriteEngine On
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule ^([^\.]+)$ $1.php [NC,L]
</Directory>

According to the apache wiki it is totally equivalent to a .htaccess file.

I am using this in the main config: ``` DocumentRoot "/srv/http/4get" <Directory "/srv/http/4get"> Options +MultiViews RewriteEngine On RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !-f RewriteRule ^([^\.]+)$ $1.php [NC,L] </Directory> ``` According to the apache wiki it is totally equivalent to a .htaccess file.
Owner

Place this under a <VirtualHost *:80> in your configuration and restart apache

Place this under a `<VirtualHost *:80>` in your configuration and restart apache
Author

I added it and as before I see Error 404.
Nothing new in the log.

I added it and as before I see `Error 404`. Nothing new in the log.
Owner

You placed this in /etc/httpd/confextra/httpd-ssl.conf?

You placed this in `/etc/httpd/confextra/httpd-ssl.conf`?
Author

No in /etc/httpd/conf/httpd.conf.
I thought that SSL is not needed for local use and /etc/httpd/confextra/httpd-ssl.conf is not included anymore:

# Secure (SSL/TLS) connections
#Include conf/extra/httpd-ssl.conf
No in `/etc/httpd/conf/httpd.conf`. I thought that SSL is not needed for local use and `/etc/httpd/confextra/httpd-ssl.conf` is not included anymore: ``` # Secure (SSL/TLS) connections #Include conf/extra/httpd-ssl.conf ```
Owner

Another person got in touch with me, and also has trouble setting this up on Arch, so this isn't a you thing. I'll investigate later on and write a guide on how to fix this mess.

Another person got in touch with me, and also has trouble setting this up on Arch, so this isn't a you thing. I'll investigate later on and write a guide on how to fix this mess.

Chiming in, it's me the other person ;) I think that the best solution for Arch should be using a docker container. However, in every case, we have to face the same problem: if we selfhost locally on our machine, Im afraid that searching from http://localhost will happily serve our IP to the search engines, and that would obviously defeat the whole purpose of a private search. I have opened an issue to see whether it's possible to run a 4get private docker instance together with Tor.

Chiming in, it's me the other person ;) I think that the best solution for Arch should be using a docker container. However, in every case, we have to face the same problem: if we selfhost locally on our machine, Im afraid that searching from http://localhost will happily serve our IP to the search engines, and that would obviously defeat the whole purpose of a private search. I have opened an issue to see whether it's possible to run a 4get private docker instance together with Tor.
Author

Possibly I'd run it without a docker but if you need it for Tor or something I guess I'll use it.

I will wait for the guide then, thanks.

Possibly I'd run it without a docker but if you need it for Tor or something I guess I'll use it. I will wait for the guide then, thanks.

@root actually if you use Tor you don't need to run 4get on docker, you just have to edit two files in the data folder and 4get will route the traffic to the Tor proxy. If you, like me, run a docker container to get rid of the lack of dependencies, you'll have to add a Tor docker image to docker-compose.yml. But here you're on your own unless a guide will be ready, there's no official Tor docker image and the documentation on how to run Tor together with a search engine on docker is very sparse.
Also, keep in mind that running 4get over Tor or a vpn would have Google blocking you very soon, and there's no way to circumvent this.

@root actually if you use Tor you don't need to run 4get on docker, you just have to edit two files in the data folder and 4get will route the traffic to the Tor proxy. If you, like me, run a docker container to get rid of the lack of dependencies, you'll have to add a Tor docker image to docker-compose.yml. But here you're on your own unless a guide will be ready, there's no official Tor docker image and the documentation on how to run Tor together with a search engine on docker is very sparse. Also, keep in mind that running 4get over Tor or a vpn would have Google blocking you very soon, and there's no way to circumvent this.
Owner

Also, keep in mind that running 4get over Tor or a vpn would have Google blocking you very soon, and there's no way to circumvent this.

You can setup proxychains

>Also, keep in mind that running 4get over Tor or a vpn would have Google blocking you very soon, and there's no way to circumvent this. You can setup proxychains

Great, I have just found out a brand new docker-tor guide, amazing surprise and I immediately followed it, thank you guys! :) I wonder how would it be possible to use proxychains to launch the browser. The proxychains.conf has socks4 127.0.0.1 9050 as default proxy, it looks like it needs Tor running on the host. I wonder how this could work with 4get running over Tor in an isolated container.

Great, I have just found out a brand new docker-tor guide, amazing surprise and I immediately followed it, thank you guys! :) I wonder how would it be possible to use proxychains to launch the browser. The proxychains.conf has socks4 127.0.0.1 9050 as default proxy, it looks like it needs Tor running on the host. I wonder how this could work with 4get running over Tor in an isolated container.
Author

Usually I scrape DuckDuckGO so I think that I can just use docker with tor.
When/if somebody makes a guide can you let me know?

Usually I scrape DuckDuckGO so I think that I can just use docker with tor. When/if somebody makes a guide can you let me know?
Hey @root the guide is on https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/docker_tor.md
Author

Thanks @Evhorizon.

Do I need to follow https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/docker.md before and than that?

Thanks @Evhorizon. Do I need to follow https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/docker.md before and than that?
Owner

@root yes

@root yes
Author

Simply running docker compose up -d in 4get repo (/srv/http/4get here) works.
I can use 4get connecting to http://localhost/ and search.

After I tried to add tor.
I made the files in /srv/http/4get/data and /srv/http/4get/proxies.

I am not sure to how make the folders.

If I run docker compose up -d in .../4get/data and in .../4get/proxies 4get shows up but I cannot search.

Simply running `docker compose up -d` in 4get repo (`/srv/http/4get` here) works. I can use 4get connecting to `http://localhost/` and search. After I tried to add tor. I made the files in `/srv/http/4get/data` and `/srv/http/4get/proxies`. I am not sure to how make the folders. If I run `docker compose up -d` in `.../4get/data` and in `.../4get/proxies` 4get shows up but I cannot search.

I think the easy way should be creating a 4get folder in your home directory and there create 1) a "proxies" folder with the suggested onion.txt file and 2) the docker-compose.yml file. Then cd to this 4get folder and run docker compose up -d. It works for me.

I think the easy way should be creating a 4get folder in your home directory and there create 1) a "proxies" folder with the suggested onion.txt file and 2) the docker-compose.yml file. Then cd to this 4get folder and run docker compose up -d. It works for me.
Author

In this 4get folder is docker-compose.yaml inside proxies folder?

Where do I put torrc and other docker-compose.yaml in /srv/http/4get?

In this 4get folder is `docker-compose.yaml` inside proxies folder? Where do I put torrc and other `docker-compose.yaml` in `/srv/http/4get`?

No, you have this 4get folder with two distinct things, a proxies subfolder and the docker compose file. Do you want to use your own torrc file ? The Tor container will already create a default torrc file.

No, you have this 4get folder with two distinct things, a proxies subfolder and the docker compose file. Do you want to use your own torrc file ? The Tor container will already create a default torrc file.
Author

No, I do not have any torrc file and I do not have experience with this.

Where do I create the torrc file?

No, I do not have any torrc file and I do not have experience with this. Where do I create the torrc file?

Ok so you don't need to create it, just run docker compose and everything will work out of the box

Ok so you don't need to create it, just run docker compose and everything will work out of the box
Author

It fails to start:

mount /path/4get/torrc:/etc/tor/torrc (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Indeed /etc/tor/torrc is a text file and 4get/torrc is a folder.
I removed that line (- ./torrc/torrc:/etc/tor/torrc) and localhost is completely down.

It fails to start: ``` mount /path/4get/torrc:/etc/tor/torrc (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type ``` Indeed `/etc/tor/torrc` is a text file and 4get/torrc is a folder. I removed that line (`- ./torrc/torrc:/etc/tor/torrc`) and localhost is completely down.

Hmm weird, can you explain in details how did you setup the docker container? Have you removed your previous containers and images before starting this one?

Hmm weird, can you explain in details how did you setup the docker container? Have you removed your previous containers and images before starting this one?
Author

Do I need to download the images manually? What's luuul?

Do I need to download the images manually? What's `luuul`?

No you don't need it, the docker compose file will do all the job for you. Luuul is the name of the docker repo with the two images (Tor and 4get). Remove all the images and containers that you have and start from scratch following exactly what the guide says.

No you don't need it, the docker compose file will do all the job for you. Luuul is the name of the docker repo with the two images (Tor and 4get). Remove all the images and containers that you have and start from scratch following exactly what the guide says.
Author

The guide is not clear.

lsd --tree
 ./
├──  docker-compose.yaml
└──  proxies/
    ├──  docker-compose.yaml
    └──  onion.txt

Like that?

Same error as before starting it creates a torrc folder.

The guide is not clear. ``` lsd --tree  ./ ├──  docker-compose.yaml └──  proxies/ ├──  docker-compose.yaml └──  onion.txt ``` Like that? Same error as before starting it creates a torrc folder.

No wait, remove the docker compose file from the proxies folder. You just need the first one. Just move to that lsd folder and type docker compose up -d

No wait, remove the docker compose file from the proxies folder. You just need the first one. Just move to that lsd folder and type docker compose up -d
Author

There are two docker-compose.yaml in the guide I am confused.

I moved the one in proxies and as before I ran docker compose up -d in the parent folder.
Same error as before starting it creates a torrc folder.

The folder is named 4get lsd is a command: Modern ls with a lot of pretty colors and awesome icons.

There are two `docker-compose.yaml` in the guide I am confused. I moved the one in proxies and as before I ran `docker compose up -d` in the parent folder. Same error as before starting it creates a torrc folder. The folder is named 4get `lsd` is a command: `Modern ls with a lot of pretty colors and awesome icons`.

Oh sorry, I have always just used ls :) the first docker compose file is to route searches over Tor, the second is to be found as a hidden service by Tor users, so I guess you would only want the first one as me

Oh sorry, I have always just used ls :) the first docker compose file is to route searches over Tor, the second is to be found as a hidden service by Tor users, so I guess you would only want the first one as me
Author

I use librewolf and I have this error:

Secure Connection Failed

An error occurred during a connection to localhost:80. SSL received a record that exceeded the maximum permissible length.

Error code: SSL_ERROR_RX_RECORD_TOO_LONG

https://support.mozilla.org/en-US/kb/secure-connection-failed-firefox-did-not-connect?as=u#w_secure-connection-failed

There is no option to add a security exception to bypass this type of error.

I use librewolf and I have this error: ``` Secure Connection Failed An error occurred during a connection to localhost:80. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG ``` https://support.mozilla.org/en-US/kb/secure-connection-failed-firefox-did-not-connect?as=u#w_secure-connection-failed >There is no option to add a security exception to bypass this type of error.
Owner

Why did you setup SSL over localhost?

Why did you setup SSL over localhost?
Author

I do not know why I have it.
How can I disable it?

I do not know why I have it. How can I disable it?

@root dumb question, are you sure you are typing http://localhost/ and not https://localhost/ ?

@root dumb question, are you sure you are typing http://localhost/ and not https://localhost/ ?
Author

I was not using http sorry.

I have this error:

Shit
This scraper returned an error:
Failed to get html
I was not using http sorry. I have this error: ``` Shit This scraper returned an error: Failed to get html ```

@root can you paste the output of docker container ls? And the content of the onion.txt in the proxies folder? It looks like something is still misconfigured.

@root can you paste the output of `docker container ls`? And the content of the onion.txt in the proxies folder? It looks like something is still misconfigured.
Author
docker container ls
CONTAINER ID   IMAGE               COMMAND                  CREATED          STATUS                             PORTS                                        NAMES
acf0e6039afa   luuul/4get:latest   "./docker/docker-ent…"   40 seconds ago   Up 39 seconds                      0.0.0.0:80->80/tcp, :::80->80/tcp, 443/tcp   4get-fourget-1
1b579790132e   luuul/tor:latest    "/usr/bin/tor"           40 seconds ago   Up 39 seconds (health: starting)   9050/tcp                                     4get-tor-1
88e8598b2706   luuul/4get:latest   "./docker/docker-ent…"   25 hours ago     Up 23 hours                        80/tcp, 443/tcp                              data-fourget-1
cat proxies/onion.txt 
# proxies/onion.txt
# Note: "tor" is the service name of luuul/tor in docker-compose.yaml
socks5:tor:9050::
``` docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES acf0e6039afa luuul/4get:latest "./docker/docker-ent…" 40 seconds ago Up 39 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp, 443/tcp 4get-fourget-1 1b579790132e luuul/tor:latest "/usr/bin/tor" 40 seconds ago Up 39 seconds (health: starting) 9050/tcp 4get-tor-1 88e8598b2706 luuul/4get:latest "./docker/docker-ent…" 25 hours ago Up 23 hours 80/tcp, 443/tcp data-fourget-1 ``` ``` cat proxies/onion.txt # proxies/onion.txt # Note: "tor" is the service name of luuul/tor in docker-compose.yaml socks5:tor:9050:: ```
Owner

I was not using http sorry.

I have this error:

Shit
This scraper returned an error:
Failed to get html

does this error appear on every engine? If so, that might be an issue with your DNS resolver.

> I was not using http sorry. > > I have this error: > ``` > Shit > This scraper returned an error: > Failed to get html > ``` > does this error appear on every engine? If so, that might be an issue with your DNS resolver.
Author

@lolcat I tested it with 4get.ca and it works e.g. https://4get.ca/web?s=test.
Before and now too.

@lolcat I tested it with 4get.ca and it works e.g. `https://4get.ca/web?s=test`. Before and now too.

@root you have two running docker 4get containers.Try to delete the last one with docker container rm -f 88e8598b2706. If this doesn't work, try to remove all the containers and images that you have and start all the setup from scratch.

@root you have two running docker 4get containers.Try to delete the last one with docker container rm -f 88e8598b2706. If this doesn't work, try to remove all the containers and images that you have and start all the setup from scratch.
Owner

@lolcat I tested it with 4get.ca and it works e.g. https://4get.ca/web?s=test.
Before and now too.

No I don't mean on 4get.ca, I mean on your own config. Do all of the scrapers fail?

> @lolcat I tested it with 4get.ca and it works e.g. `https://4get.ca/web?s=test`. > Before and now too. No I don't mean on 4get.ca, I mean on your own config. Do all of the scrapers fail?
Author

In /srv/http/4get/ I just ran chmod 777 -R icons/.

In my home in 4get I have only:

lsd --tree 
 ./
├──  docker-compose.yaml
└──  proxies/
    └──  onion.txt
cat docker-compose.yaml 
# docker-compose.yaml
version: "3.7"

...      


    depends_on:
     - tor
     
    volumes:
      - ./proxies/:/srv/http/4get/data/proxies/

No I don't mean on 4get.ca, I mean on your own config. Do all of the scrapers fail?

Also yandex, brave , curlie, marginalia and qwant.

In `/srv/http/4get/` I just ran `chmod 777 -R icons/`. In my home in 4get I have only: ``` lsd --tree  ./ ├──  docker-compose.yaml └──  proxies/ └──  onion.txt ``` ``` cat docker-compose.yaml # docker-compose.yaml version: "3.7" ... depends_on: - tor volumes: - ./proxies/:/srv/http/4get/data/proxies/ ``` >No I don't mean on 4get.ca, I mean on your own config. Do all of the scrapers fail? Also yandex, brave , curlie, marginalia and qwant.
Author
docker rmi -f $(docker images -aq)
docker rmi -f $(docker images -aq)
docker system prune -a

After that still Failed to get html.

In /etc/tor/torrc I added SocksPort 0.0.0.0:9050.

``` docker rmi -f $(docker images -aq) docker rmi -f $(docker images -aq) docker system prune -a ``` After that still `Failed to get html`. In `/etc/tor/torrc` I added `SocksPort 0.0.0.0:9050`.

@root Just wonder, how are your librewolf network settings?

@root Just wonder, how are your librewolf network settings?
Collaborator

It looks like you're mixing the steps from your previous install attempt and the steps from docker
please don't put SocksPort 0.0.0.0:9050 in your host's /etc/tor/torrc

In my home in 4get I have only:

lsd --tree 
 ./
├──  docker-compose.yaml
└──  proxies/
    └──  onion.txt

This is good! The files you have in your home folder are all you need if you're using the docker + tor guide. However the contents in your docker compose seems incomplete,

Put the following in your docker-compose.yaml file in your home's 4get folder

# docker-compose.yaml
version: "3.7"

services:
  tor:
    image: luuul/tor:latest
    restart: unless-stopped
    # Warning: Do not publish port 9050
    
  fourget:
    image: luuul/4get:latest
    restart: unless-stopped
    environment:
      - FOURGET_PROTO=http
      - FOURGET_SERVER_NAME=4get.ca
      # loads proxies/onion.txt
      - FOURGET_PROXY_DDG="onion" 
      - FOURGET_PROXY_BRAVE="onion"
      - FOURGET_PROXY_FB="onion"
      - FOURGET_PROXY_GOOGLE="onion"
      - FOURGET_PROXY_QWANT="onion"
      - FOURGET_PROXY_MARGINALIA="onion"
      - FOURGET_PROXY_MOJEEK="onion"
      - FOURGET_PROXY_SC="onion"
      - FOURGET_PROXY_SPOTIFY="onion"
      - FOURGET_PROXY_WIBY="onion"
      - FOURGET_PROXY_CURLIE="onion"
      - FOURGET_PROXY_YT="onion"
      - FOURGET_PROXY_YEP="onion"
      - FOURGET_PROXY_PINTEREST="onion"
      - FOURGET_PROXY_SEZNAM="onion"
      - FOURGET_PROXY_NAVER="onion"
      - FOURGET_PROXY_GREPPR="onion"
      - FOURGET_PROXY_CROWDVIEW="onion"
      - FOURGET_PROXY_MWMBL="onion"
      - FOURGET_PROXY_FTM="onion"
      - FOURGET_PROXY_IMGUR="onion"
      - FOURGET_PROXY_YANDEX_W="onion"
      - FOURGET_PROXY_YANDEX_I="onion"
      - FOURGET_PROXY_YANDEX_V="onion"

    ports:
      - "80:80"
      
    depends_on:
     - tor
     
    volumes:
      - ./proxies/:/var/www/html/4get/data/proxies/

then cd into that directory and run docker compose up -d
everything should work

If there are issues please give the result of these commands

docker ps -a 
docker container ls -a
It looks like you're mixing the steps from your previous install attempt and the steps from docker please don't put `SocksPort 0.0.0.0:9050` in your host's /etc/tor/torrc > In my home in 4get I have only: ``` lsd --tree  ./ ├──  docker-compose.yaml └──  proxies/ └──  onion.txt ``` This is good! The files you have in your home folder are all you need if you're using the docker + tor guide. However the contents in your docker compose seems incomplete, Put the following in your docker-compose.yaml file in your home's 4get folder ``` # docker-compose.yaml version: "3.7" services: tor: image: luuul/tor:latest restart: unless-stopped # Warning: Do not publish port 9050 fourget: image: luuul/4get:latest restart: unless-stopped environment: - FOURGET_PROTO=http - FOURGET_SERVER_NAME=4get.ca # loads proxies/onion.txt - FOURGET_PROXY_DDG="onion" - FOURGET_PROXY_BRAVE="onion" - FOURGET_PROXY_FB="onion" - FOURGET_PROXY_GOOGLE="onion" - FOURGET_PROXY_QWANT="onion" - FOURGET_PROXY_MARGINALIA="onion" - FOURGET_PROXY_MOJEEK="onion" - FOURGET_PROXY_SC="onion" - FOURGET_PROXY_SPOTIFY="onion" - FOURGET_PROXY_WIBY="onion" - FOURGET_PROXY_CURLIE="onion" - FOURGET_PROXY_YT="onion" - FOURGET_PROXY_YEP="onion" - FOURGET_PROXY_PINTEREST="onion" - FOURGET_PROXY_SEZNAM="onion" - FOURGET_PROXY_NAVER="onion" - FOURGET_PROXY_GREPPR="onion" - FOURGET_PROXY_CROWDVIEW="onion" - FOURGET_PROXY_MWMBL="onion" - FOURGET_PROXY_FTM="onion" - FOURGET_PROXY_IMGUR="onion" - FOURGET_PROXY_YANDEX_W="onion" - FOURGET_PROXY_YANDEX_I="onion" - FOURGET_PROXY_YANDEX_V="onion" ports: - "80:80" depends_on: - tor volumes: - ./proxies/:/var/www/html/4get/data/proxies/ ``` then cd into that directory and run docker compose up -d everything should work If there are issues please give the result of these commands ``` docker ps -a docker container ls -a ```
Author

@Evhorizon I think that I did not change network settings.
I just tried firefox same there.

@throwaway Shouldn't I change - ./proxies/:/var/www/html/4get/data/proxies to - ./proxies/:/srv/http/4get/data/proxies/ since 4get repo is there?

I put only the beginning and the part that I changed with ... in between in that file.

docker ps -a
CONTAINER ID   IMAGE               COMMAND                  CREATED          STATUS                             PORTS                                        NAMES
eaa119314b01   luuul/4get:latest   "./docker/docker-ent…"   11 seconds ago   Up 10 seconds                      0.0.0.0:80->80/tcp, :::80->80/tcp, 443/tcp   4get-fourget-1
7ad8dfbb4b06   luuul/tor:latest    "/usr/bin/tor"           11 seconds ago   Up 10 seconds (health: starting)   9050/tcp    

docker container ls -a
CONTAINER ID   IMAGE               COMMAND                  CREATED          STATUS                             PORTS                                        NAMES
eaa119314b01   luuul/4get:latest   "./docker/docker-ent…"   27 seconds ago   Up 26 seconds                      0.0.0.0:80->80/tcp, :::80->80/tcp, 443/tcp   4get-fourget-1
7ad8dfbb4b06   luuul/tor:latest    "/usr/bin/tor"           27 seconds ago   Up 26 seconds (health: starting)   9050/tcp   

I reset /etc/tor/torrc no need to run tor service?
With it running or not curlie and DDG scraper: Failed to get html.

@Evhorizon I think that I did not change network settings. I just tried `firefox` same there. @throwaway Shouldn't I change `- ./proxies/:/var/www/html/4get/data/proxies` to `- ./proxies/:/srv/http/4get/data/proxies/` since 4get repo is there? I put only the beginning and the part that I changed with `...` in between in that file. ``` docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES eaa119314b01 luuul/4get:latest "./docker/docker-ent…" 11 seconds ago Up 10 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp, 443/tcp 4get-fourget-1 7ad8dfbb4b06 luuul/tor:latest "/usr/bin/tor" 11 seconds ago Up 10 seconds (health: starting) 9050/tcp docker container ls -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES eaa119314b01 luuul/4get:latest "./docker/docker-ent…" 27 seconds ago Up 26 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp, 443/tcp 4get-fourget-1 7ad8dfbb4b06 luuul/tor:latest "/usr/bin/tor" 27 seconds ago Up 26 seconds (health: starting) 9050/tcp ``` I reset `/etc/tor/torrc` no need to run tor service? With it running or not curlie and DDG scraper: `Failed to get html`.
Collaborator

Shouldn't I change - ./proxies/:/var/www/html/4get/data/proxies to - ./proxies/:/srv/http/4get/data/proxies/ since 4get repo is there?

In docker, volumes are mounted from the host machine to the container

the 4get container always reads from /var/www/html/4get/data/proxies so that part will not change
the ./proxies/ is relative to where you're running docker compose. If you'd like you can change this part to /srv/http/4get/data/proxies/

so the volume mount could look like

- /srv/http/4get/data/proxies/:/var/www/html/4get/data/proxies

but since your proxy list onion.txt is in the 4get folder in your home directory you don't need to do this.

I reset /etc/tor/torrc no need to run tor service?

Yes. The tor service will run inside the tor container luuul/tor which has a working torrc inside

I put only the beginning and the part that I changed with ... in between in that file.

Okay. Please make sure that your docker-compose.yaml contains only the content I mentioned previously


Docker compose will restart already existing containers if they exist. And containers contain their own state. We should try starting with a blank state and recreate our containers.

stop all containers with docker container stop $(docker container ls -aq)
remove containers with docker container rm $(docker container ls -aq)

then try docker compose up which will recreate the containers

> Shouldn't I change - ./proxies/:/var/www/html/4get/data/proxies to - ./proxies/:/srv/http/4get/data/proxies/ since 4get repo is there? In docker, volumes are mounted from the host machine to the container the 4get container always reads from /var/www/html/4get/data/proxies so that part will not change the `./proxies/` is relative to where you're running docker compose. If you'd like you can change this part to `/srv/http/4get/data/proxies/` so the volume mount could look like ``` - /srv/http/4get/data/proxies/:/var/www/html/4get/data/proxies ``` but since your proxy list `onion.txt` is in the 4get folder in your home directory you don't need to do this. >I reset /etc/tor/torrc no need to run tor service? Yes. The tor service will run inside the tor container `luuul/tor` which has a working torrc inside >I put only the beginning and the part that I changed with ... in between in that file. Okay. Please make sure that your docker-compose.yaml contains only the content I mentioned previously --- Docker compose will restart already existing containers if they exist. And containers contain their own state. We should try starting with a blank state and recreate our containers. stop all containers with ` docker container stop $(docker container ls -aq)` remove containers with `docker container rm $(docker container ls -aq)` then try docker compose up which will recreate the containers
Author

The cloned 4get repo in /srv/http/ is totally pointless so.

It works thanks.
I could make a version of https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/docker_tor.md for local use.

I only changed FOURGET_SERVER_NAME so that I can distinguish 4get.ca and the local instance when I look the search engines in the browser.

The cloned 4get repo in `/srv/http/` is totally pointless so. It works thanks. I could make a version of https://git.lolcat.ca/lolcat/4get/src/branch/master/docs/docker_tor.md for local use. I only changed `FOURGET_SERVER_NAME` so that I can distinguish 4get.ca and the local instance when I look the search engines in the browser.
root closed this issue 2024-06-25 14:38:44 +00:00
Sign in to join this conversation.
No Label
No Milestone
No project
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: lolcat/4get#19
No description provided.