Project

General

Profile

Actions

Bug #1221

closed

HTTP 429 response on SearxNG searches

Added by José Manuel García-Patos 20 days ago. Updated 17 days ago.

Status:
Closed
Priority:
1.x
Start date:
09/15/2024
Due date:
% Done:

0%

Estimated time:

Description

When trying to search for anything on SearXNG instances, the server returns an HTTP 429 response and the text “Too Many Requests”.

Looking at their code (https://github.com/searxng/searxng/tree/master/searx/botdetection), the reason seems to be that Privacy Browser PC doesn't send certain headers that SearXNG expects in order to distinguish between a bot and a legitimate user. The headers in question are:

  • Accept-Encoding: any string containing ‘gzip’, ‘deflate’ or both will do, but of course a spec-wise correct value would be much better.
  • Accept-Language: its mere presence will do. I tested it using ‘*’ and it worked.
  • Connection: it must be sent in the request and its value MUST NOT be ‘close’. (I tested it with ‘keep-alive’ personally.)

There's another header that's checked (‘User-Agent’), but here Privacy Browser PC complies.

Incidentally, this doesn't happen with the Android version.


Files

UrlRequestInterceptor.diff (1.06 KB) UrlRequestInterceptor.diff Diff file for the class UrlRequestInterceptor José Manuel García-Patos, 09/17/2024 10:53 AM
Actions #1

Updated by Soren Stoutner 19 days ago

  • Assignee set to Soren Stoutner

The root cause of this is likely that the Qt 5 WebEngine is quite old from a feature perspective, as documented in the Handbook inside of Privacy Browser PC and at:

https://wiki.qt.io/QtWebEngine/ChromiumVersions

This will be resolved when Privacy Browser PC switches to Qt 6 WebEngine, which should happen shortly when KDE 6 lands in Debian testing.

Feature #1223: Migrate to Qt 6

As a side note, it appears that the current Qt WebEngine sends the Accept-Encoding header, but not the Accept-Language header.

https://browserleaks.com/ip

Actions #2

Updated by José Manuel García-Patos 18 days ago

I patched the code of version 0.6.1 as in the attached diff file, tested it and it worked. I'm not a C++ programmer, so probably that's not the right way to do it, but it may be helpful to you somehow.

Actions #3

Updated by Soren Stoutner 18 days ago

Thanks for providing the patch. Before implementing it, would you be willing to sign a Code Submission Agreement as described at:

https://www.stoutner.com/privacy-browser-pc/contributors/

https://download.stoutner.com/Stoutner%20Code%20Submission%20Agreement.pdf

You can send the completed form directly to my email address at .

Actions #4

Updated by José Manuel García-Patos 17 days ago

Soren Stoutner wrote in #note-3:

Thanks for providing the patch.

I don't really consider it a patch. It's more like the starting point for one. For example, the values of some or all of those headers (at the very least 'Accept-Language') should come from the user's configuration. Also, those assignments would be best done in their own method, not in interceptRequest itself, in case more headers needed to be added later. Finally, I'm not a C++ programmer. I just browsed through the QWebEngine documentation and tried something that happened to work.

I guess this is a roundabout way of saying that I wouldn't use that code as-is myself, so there's no reason to sign any contract. I'm sure you can do much better than me, and I can't see how I could claim any copyright on code written by you that may have, or may have not, been loosely based on a few lines I informally gave you as an example. If you want that code, it's yours. (I hope I'm not being rude by refusing. It's not my intention.)

Actions #5

Updated by Soren Stoutner 17 days ago

  • Status changed from New to Closed

That's OK. I'll probably just close this then, as it will be fixed by the update to Qt 6 WebEngine.

Actions

Also available in: Atom PDF