Is it possible for a website to detect the presence of a proxy?
I am attempting to scrape a site which is notoriously difficult to scrape. Access from datacentres is generally blocked. In the past I've used various proxies, but recently these have stopped working.
The site employs various pitfalls when it doesn't like the user; e.g. certain javascript components fail, or the server redirects AJAX requests to localhost; thus causing a null-route.
I had previously assumed that the server was filtering by IP -- Recently I've noticed that the site acts up even from a "good" IP address, but only if proxied. In other words, if I open the site from a browser in computer A, it works perfectly fine. If I try to connect from computer B which uses computer A as a proxy server, the site fails to load. Even if I connect from computer A using the proxy server running on itself, the site still fails to load.
Which leads me to believe that the site is somehow detecting the existence of a proxy.
The proxy software is one I've written myself, so I know for certain that it does not add any headers which would give it away. I have used it successfully for many years without issue, so it's unlikely to have an obvious bug. It cannot be queried by the remote server. It doesn't mess with the headers or certificates -- it only forwards https traffic with the CONNECT method. (There is no HTTP traffic)
The browser I'm using is Firefox, and WebRTC is disabled.
My question is: is there any way for a website/webserver to detect:
- That a browser has some proxy settings configured?
- That a proxy server is being used at all
