Offline Website grabbers / broswers
Offline Website grabbers / broswers
I have noticed a couple of people using website grabbers (ie programs which download the WHOLE of a site) over the past few months. I was letting it go because while they were using a good bit of bandwidth, I still had lots to spare but now it's getting silly.
It's only 10 days into February and one guy from 217.43.159.22 who's using a website grabber has used 12.25% of the website bandwidth so far. He's downloaded over 200Mb of files in only 4 visits!!!
It pisses me off that there's always a couple of people who try to abuse things and make it worse for other people (ie, probably slowing down the site).
Now, because of this guy and probably a few others, I have to waste time and figure out a way to stop these website grabbers eating all my bandwidth.
Die leechers die! :-E
[update]
Just noticed that the same guy has used 5.77% of the bandwidth of my other site, which is used to host some of the issue scans
[/update]
It's only 10 days into February and one guy from 217.43.159.22 who's using a website grabber has used 12.25% of the website bandwidth so far. He's downloaded over 200Mb of files in only 4 visits!!!
It pisses me off that there's always a couple of people who try to abuse things and make it worse for other people (ie, probably slowing down the site).
Now, because of this guy and probably a few others, I have to waste time and figure out a way to stop these website grabbers eating all my bandwidth.
Die leechers die! :-E
[update]
Just noticed that the same guy has used 5.77% of the bandwidth of my other site, which is used to host some of the issue scans
[/update]
Damn!! It took until now (2 hours) to get it working correctly. I should be a bit of an expert of robot.txt and .htacces files now!!
This should stop this guy.. at least for now, I'll have to add more barriers later.
The Internet is really like the wild west these days
Let me know if I've broken anything on the site with these access changes!!
This should stop this guy.. at least for now, I'll have to add more barriers later.
The Internet is really like the wild west these days
Let me know if I've broken anything on the site with these access changes!!
- Lloyd Mangram
- King of Ludlow
- Posts: 1152
- Joined: Thu Jun 19, 2003 10:22 pm
- Location: Ludlow
- Contact:
Oh dear....
I wanted to ask your permission first, and later I wanted to inform you about it but forgot, but errr... I have to admit I 'mirrored' your whole site
AFTER the forum hijack, unfortunately!
I thought if anything would happen in the future I would have a backup at least, preserving all your hard work if we'd ever run into trouble (again).
Of course, I had no intentions to steal your bandwidth or anything, it was just a one timer for me, only to occasionally upgrade the backup if needed.
(with all the evil scripts you have running now, I can forget that now )
So, I'm really sorry about it all! Don't ban me!
Ta!
Rob
I wanted to ask your permission first, and later I wanted to inform you about it but forgot, but errr... I have to admit I 'mirrored' your whole site
AFTER the forum hijack, unfortunately!
I thought if anything would happen in the future I would have a backup at least, preserving all your hard work if we'd ever run into trouble (again).
Of course, I had no intentions to steal your bandwidth or anything, it was just a one timer for me, only to occasionally upgrade the backup if needed.
(with all the evil scripts you have running now, I can forget that now )
So, I'm really sorry about it all! Don't ban me!
Ta!
Rob
Once again I emerge from beneath a massive pile of paper which makes my desk groan to bring you the world’s most amazing posts.
On a separate issue, I noticed that there was '32 people on-line' yesterday when I was logged in.
Contributor to Def Tribute to ZZAP!
Compilation64 - http://compilation64.zzap64.co.uk/
Crazy about the C64? - www.lemon64.com
Amiga Anguish? - www.lemonamiga.com
Compilation64 - http://compilation64.zzap64.co.uk/
Crazy about the C64? - www.lemon64.com
Amiga Anguish? - www.lemonamiga.com
- Lloyd Mangram
- King of Ludlow
- Posts: 1152
- Joined: Thu Jun 19, 2003 10:22 pm
- Location: Ludlow
- Contact:
Hi Iain, don't hate me.iain wrote:Rob!!!!!
You should have let me know before!! But... you only did it once? And how long ago was that? Were you using Httracter?
The above program has been taking about 10% of my bandwidth for the past few months though...
I did it with Httracker, (excellent proggie, btw) in a couple of sessions.
I back-upped the whole site, including the forum, but NOT THE SCANS! (of course, and why should I, I have them all on cd/hd)
You can choose which folders you want to 'mirror' and I excluded, or killed, all folders with scans. Occasionally I seemed to have mirrored some other stuff as well on the same server. (tfyatdt-stuff)
It was in the week (or two weeks ,max) after the Rrap got hacked, and yesss..I only did it ONCE.
I'm really, really sorry.
I never meant to get you into any sort of trouble as I really appreciate what you are doing. I'm sure you're aware of that fact.
Hugs?
Rob
Once again I emerge from beneath a massive pile of paper which makes my desk groan to bring you the world’s most amazing posts.
Well, I didn't get into any trouble, I'm not over my bandwidth limit but I don't want lots of people grabbing the whole site lots though.
I forgive you this time Rob! But don't worry, there's other people doing it as well.
By modifying the .htacess file I should be able to stop a lot of email harvesters etc. so the extra knowledge I have gained nwill be usefull.
I forgive you this time Rob! But don't worry, there's other people doing it as well.
By modifying the .htacess file I should be able to stop a lot of email harvesters etc. so the extra knowledge I have gained nwill be usefull.
- Lloyd Mangram
- King of Ludlow
- Posts: 1152
- Joined: Thu Jun 19, 2003 10:22 pm
- Location: Ludlow
- Contact:
I don't fucking believe it! The website is getting hammered again, prob by offoline website grabbers, 17,000 pages have been served already today (the avg for a full day is 3000-4000).
My months bandwidth is almost up, if this continues I'll have to shut down the site for a day or two!!!
So much hassle!!!! :-E
My months bandwidth is almost up, if this continues I'll have to shut down the site for a day or two!!!
So much hassle!!!! :-E
Look at all the bandwidth these fuckers have used. You can see the crazy spikes of bandwidth usage at the beginning of this month and again today.
Anyway, I have banned the current ones IP now (couldn't do it from college earlier!)
I'll have to do some serious anti crawler etc. coding for the site after my exams!
Anyway, I have banned the current ones IP now (couldn't do it from college earlier!)
I'll have to do some serious anti crawler etc. coding for the site after my exams!
Unbelievable, the website is under attack again from the 212.69.226.40 gobshite at legend.co.uk. Guess I didn't implement my ban correctly!
The worm that's trying to hack phpBB forums must also be all over the place, I've had 11609 forbidden access today as well!
The Internet is just like a war zone at the moment, it's sad how it's gotten really, compared to how it used to be back when I started using it in 1994
The joys of running a website!
The worm that's trying to hack phpBB forums must also be all over the place, I've had 11609 forbidden access today as well!
The Internet is just like a war zone at the moment, it's sad how it's gotten really, compared to how it used to be back when I started using it in 1994
The joys of running a website!
-
- King of Zzap Towers
- Posts: 610
- Joined: Thu Jun 19, 2003 6:45 pm
- Location: UK
- Contact: