Bug #1221
closedHTTP 429 response on SearxNG searches
0%
Description
When trying to search for anything on SearXNG instances, the server returns an HTTP 429 response and the text “Too Many Requests”.
Looking at their code (https://github.com/searxng/searxng/tree/master/searx/botdetection), the reason seems to be that Privacy Browser PC doesn't send certain headers that SearXNG expects in order to distinguish between a bot and a legitimate user. The headers in question are:
- Accept-Encoding: any string containing ‘gzip’, ‘deflate’ or both will do, but of course a spec-wise correct value would be much better.
- Accept-Language: its mere presence will do. I tested it using ‘*’ and it worked.
- Connection: it must be sent in the request and its value MUST NOT be ‘close’. (I tested it with ‘keep-alive’ personally.)
There's another header that's checked (‘User-Agent’), but here Privacy Browser PC complies.
Incidentally, this doesn't happen with the Android version.
Files
Updated by Soren Stoutner 3 months ago
- Assignee set to Soren Stoutner
The root cause of this is likely that the Qt 5 WebEngine is quite old from a feature perspective, as documented in the Handbook inside of Privacy Browser PC and at:
https://wiki.qt.io/QtWebEngine/ChromiumVersions
This will be resolved when Privacy Browser PC switches to Qt 6 WebEngine, which should happen shortly when KDE 6 lands in Debian testing.
Feature #1223: Migrate to Qt 6
As a side note, it appears that the current Qt WebEngine sends the Accept-Encoding header, but not the Accept-Language header.
Updated by José Manuel García-Patos 3 months ago
I patched the code of version 0.6.1 as in the attached diff file, tested it and it worked. I'm not a C++ programmer, so probably that's not the right way to do it, but it may be helpful to you somehow.
Updated by Soren Stoutner 3 months ago
Thanks for providing the patch. Before implementing it, would you be willing to sign a Code Submission Agreement as described at:
https://www.stoutner.com/privacy-browser-pc/contributors/
https://download.stoutner.com/Stoutner%20Code%20Submission%20Agreement.pdf
You can send the completed form directly to my email address at soren@stoutner.com.
Updated by José Manuel García-Patos 3 months ago
Soren Stoutner wrote in #note-3:
Thanks for providing the patch.
I don't really consider it a patch. It's more like the starting point for one. For example, the values of some or all of those headers (at the very least 'Accept-Language') should come from the user's configuration. Also, those assignments would be best done in their own method, not in interceptRequest itself, in case more headers needed to be added later. Finally, I'm not a C++ programmer. I just browsed through the QWebEngine documentation and tried something that happened to work.
I guess this is a roundabout way of saying that I wouldn't use that code as-is myself, so there's no reason to sign any contract. I'm sure you can do much better than me, and I can't see how I could claim any copyright on code written by you that may have, or may have not, been loosely based on a few lines I informally gave you as an example. If you want that code, it's yours. (I hope I'm not being rude by refusing. It's not my intention.)
Updated by Soren Stoutner 3 months ago
- Status changed from New to Closed
That's OK. I'll probably just close this then, as it will be fixed by the update to Qt 6 WebEngine.