Proxy list generator

There are numerous reasons for wanting to have a list of working open proxies - you want to use surf anonymously, test your website through different connections (both speed-acceptance testing and test for non-local issues comes to mind as valid reasons for doing this), and using it a voting script as described else where on this site.


wget -q http://www.samair.ru/proxy/ -O - | \
egrep '^((([0-9]{1,3}).?){4}:[0-9]+)' | \
awk -F ' ' '{print $1}' > newproxies.txt

After having fetched a new list of proxies we want to test, we need to test what proxies respond quick enough for us to use them.


The perl program below tests each proxy listed in the newproxies.txt file and prints the ones that respond to standard out - so it can be piped to a text file with the names of the ones to use. 



#!/usr/bin/perl
                                                                                
#require external modules.
use LWP::UserAgent;
require HTTP::Request;
                                                                                
#get proxylist:
open(FIL, "newproxies.txt");
@proxies = <FIL>;
close(FIL);
                                                                                
# Create a user agent object
$ua = LWP::UserAgent->new(env_proxy => 1, keep_alive => 0, timeout => 2);
                                                                                
foreach $proxy (@proxies) {
    chomp($proxy);
    $ua->proxy('http'=>'http://'.$proxy);
    #print "Connecting to http://".$proxy."..\n";
    my $req = HTTP::Request->new(GET => 'http://www.example.com/');
    my $res = $ua->request($req);
    if ($res->is_success) {
        print "$proxy\n";
    } else {
        #print "$proxy\n";
    }
}

$ perl workingproxies.pl > workingproxies.txt