the docker image is still on curl-impersonate 0.6.1 (lwthiker), which is dead and pretends to be firefox 117 with NSS. but USER_AGENT in config.php says firefox 157, so anything that compares the TLS fingerprint to the UA sees a firefox from 2023 claiming to be a current one.
this switches to the maintained fork, lexiforest/curl-impersonate:v2.3.0-alpine, set to firefox155 (the newest target it has). it also sets FOURGET_USER_AGENT to firefox 155 in the dockerfile so the UA and the fingerprint actually match. I tested it locally: the lib loads fine in php (curl 8.23.0-IMPERSONATE, boringssl, http2), and ddg, brave, yep, bing, qwant and marginalia all gave the same results as the current image.
gotchas:
ff117-style names don't exist anymore. if CURL_IMPERSONATE is set to something unknown, every curl_init() returns false and every scraper dies, so the lib and the env var have to change together
it's one libcurl-impersonate.so now (no more -ff/-chrome split), and on alpine it needs libstdc++. nss isn't needed anymore
pinned v2.3.0-alpine. the v2.4.0 tag on docker hub is a mistagged build from right before 2.3.0 and doesn't even have an alpine version
their images are amd64 only, same as the old one
not in this PR but worth a look:
docs/configure.md and docs/develop.md still say libcurl-impersonate-ff.so and firefox117/ff117. follow them with the new lib and no scraper works
scraper/facebook.php lines 56 and 75 hardcode a Firefox/115 UA instead of using config::USER_AGENT
the newest target is firefox 155 but the UA is at 157, so maybe bump them together from now on
the docker image is still on curl-impersonate 0.6.1 (lwthiker), which is dead and pretends to be firefox 117 with NSS. but USER_AGENT in config.php says firefox 157, so anything that compares the TLS fingerprint to the UA sees a firefox from 2023 claiming to be a current one.
this switches to the maintained fork, lexiforest/curl-impersonate:v2.3.0-alpine, set to firefox155 (the newest target it has). it also sets FOURGET_USER_AGENT to firefox 155 in the dockerfile so the UA and the fingerprint actually match. I tested it locally: the lib loads fine in php (curl 8.23.0-IMPERSONATE, boringssl, http2), and ddg, brave, yep, bing, qwant and marginalia all gave the same results as the current image.
gotchas:
- ff117-style names don't exist anymore. if CURL_IMPERSONATE is set to something unknown, every curl_init() returns false and every scraper dies, so the lib and the env var have to change together
- it's one libcurl-impersonate.so now (no more -ff/-chrome split), and on alpine it needs libstdc++. nss isn't needed anymore
- pinned v2.3.0-alpine. the v2.4.0 tag on docker hub is a mistagged build from right before 2.3.0 and doesn't even have an alpine version
- their images are amd64 only, same as the old one
not in this PR but worth a look:
- docs/configure.md and docs/develop.md still say libcurl-impersonate-ff.so and firefox117/ff117. follow them with the new lib and no scraper works
- scraper/facebook.php lines 56 and 75 hardcode a Firefox/115 UA instead of using config::USER_AGENT
- the newest target is firefox 155 but the UA is at 157, so maybe bump them together from now on
libstdc++: the new lib needs it at runtime. without it the preload fails and apache won't start
nss: the old lib used it for TLS. the new one has boringssl built in, so nss isn't needed anymore
- libstdc++: the new lib needs it at runtime. without it the preload fails and apache won't start
- nss: the old lib used it for TLS. the new one has boringssl built in, so nss isn't needed anymore
We both tried it and yep doesnt work. I think only chrome impersonation works fine, but every header in the entire project uses firefox-like header structure.
NSS is required because its what firefox uses. We need it otherwise the ja4 wont match. @throwaway tried again and it still returns invalid ja4s
Test the ja4 [here](https://lolcat.ca/trace?plain)
We both tried it and yep doesnt work. I think only chrome impersonation works fine, but every header in the entire project uses firefox-like header structure.
I can't reproduce that. Against lolcat.ca/trace?plain, real Firefox 158 gives t13d1517h2_8daaf6152771_3cbfd9057e0d, and so do curl_firefox155 from the image and PHP's curl_init() inside the image built from this PR. JA4 only looks at the ClientHello, and lexiforest sends Firefox's ClientHello through BoringSSL, so NSS isn't needed for it to match. If CURL_IMPERSONATE isn't set, or is set to an old name like ff117, you get t13d2712…, which might be what you saw. How did you test it: the docker image from this PR, or the .so in your own setup?
I can't reproduce that. Against lolcat.ca/trace?plain, real Firefox 158 gives t13d1517h2_8daaf6152771_3cbfd9057e0d, and so do curl_firefox155 from the image and PHP's curl_init() inside the image built from this PR. JA4 only looks at the ClientHello, and lexiforest sends Firefox's ClientHello through BoringSSL, so NSS isn't needed for it to match. If CURL_IMPERSONATE isn't set, or is set to an old name like ff117, you get t13d2712…, which might be what you saw. How did you test it: the docker image from this PR, or the .so in your own setup?
The issue is that im getting intermittent issues with the yep engine. Somehow it's able to detect me. If I use the old curl imper module, it seems to work okay.
I just found https://github.com/takielias/php-rnet. Wreq bindings for PHP. From past experience, I can say wreq is able to bypass cloudflare's bullshit much more efficiently. I'll be looking into replacing curl with wreq on the long run if the library ends up working.
The issue is that im getting intermittent issues with the yep engine. Somehow it's able to detect me. If I use the old curl imper module, it seems to work okay.
I just found [https://github.com/takielias/php-rnet](https://github.com/takielias/php-rnet). Wreq bindings for PHP. From past experience, I can say wreq is able to bypass cloudflare's bullshit much more efficiently. I'll be looking into replacing curl with wreq on the long run if the library ends up working.
I tested both images in Docker, master (ff117) and this PR (firefox155), with 160 Yep requests from another IP, using the same request the Yep scraper sends. Some of them were back to back. Every request returned 200 with results, and the two images behaved the same...
I tested both images in Docker, master (ff117) and this PR (firefox155), with 160 Yep requests from another IP, using the same request the Yep scraper sends. Some of them were back to back. Every request returned 200 with results, and the two images behaved the same...
You are not authorized to merge this pull request.
This pull request can be merged automatically.
This branch is out-of-date with the base branch
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
the docker image is still on curl-impersonate 0.6.1 (lwthiker), which is dead and pretends to be firefox 117 with NSS. but USER_AGENT in config.php says firefox 157, so anything that compares the TLS fingerprint to the UA sees a firefox from 2023 claiming to be a current one.
this switches to the maintained fork, lexiforest/curl-impersonate:v2.3.0-alpine, set to firefox155 (the newest target it has). it also sets FOURGET_USER_AGENT to firefox 155 in the dockerfile so the UA and the fingerprint actually match. I tested it locally: the lib loads fine in php (curl 8.23.0-IMPERSONATE, boringssl, http2), and ddg, brave, yep, bing, qwant and marginalia all gave the same results as the current image.
gotchas:
not in this PR but worth a look:
NSS is required because its what firefox uses. We need it otherwise the ja4 wont match. @throwaway tried again and it still returns invalid ja4s
Test the ja4 here
We both tried it and yep doesnt work. I think only chrome impersonation works fine, but every header in the entire project uses firefox-like header structure.
I can't reproduce that. Against lolcat.ca/trace?plain, real Firefox 158 gives t13d1517h2_8daaf6152771_3cbfd9057e0d, and so do curl_firefox155 from the image and PHP's curl_init() inside the image built from this PR. JA4 only looks at the ClientHello, and lexiforest sends Firefox's ClientHello through BoringSSL, so NSS isn't needed for it to match. If CURL_IMPERSONATE isn't set, or is set to an old name like ff117, you get t13d2712…, which might be what you saw. How did you test it: the docker image from this PR, or the .so in your own setup?
The issue is that im getting intermittent issues with the yep engine. Somehow it's able to detect me. If I use the old curl imper module, it seems to work okay.
I just found https://github.com/takielias/php-rnet. Wreq bindings for PHP. From past experience, I can say wreq is able to bypass cloudflare's bullshit much more efficiently. I'll be looking into replacing curl with wreq on the long run if the library ends up working.
I tested both images in Docker, master (ff117) and this PR (firefox155), with 160 Yep requests from another IP, using the same request the Yep scraper sends. Some of them were back to back. Every request returned 200 with results, and the two images behaved the same...
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.