I live in someone else’s house. No, not my parents, someone else. Don’t ask. Suffice to say, I’m extremely poor and can’t afford my own apartment, so a group living situation is my only option at this point. Anyway, this house has WiFi, which is cool. For a while we didn’t have WiFi and I had to walk two miles to the library to use the Internet.
And I don’t know what it is, but sometimes I swear, something on the network is throttling my bandwidth. I don’t know if it’s the ISP or the local gateway, but whatever it is, it’s slowing down older connections in favor of newer connections. I know this because every time I do a long download, it starts off fast and then slows to a crawl after the first five seconds or so. I won’t take this abuse. I’m determined to outsmart this throttling device – whatever it is – so that it will let me download large files again.
I thought maybe if I had the connection time out after detecting a certain amount of throttling and then start again from where the download left off, the throttling device would interpret it as a new connection, and I wouldn’t have to sit there hitting Ctrl+C and repeating the command over and over. But that’s not enough, because the fact that the next download attempt comes immediately after the previous one is a sure indicator that it’s the same connection. Essentially I need to fool the throttling bot into thinking I’m manually sitting there and attempting download after download with space between downloads. It doesn’t want me to automate the process and go do something else. It wants to waste my time. It doesn’t want me to be smart and find creative solutions to the problems it’s causing me. It wants me to suffer.
So I got to work writing a shell script that attempts a download, times out after five seconds of low bandwidth (meaning < 1 byte per second), then waits for a random number of seconds and tries again. It takes forever, but at least I don't have to sit there constantly retrying like the throttling bot wants me to do, and to be frank, it doesn't take nearly as long as it did when I wasn't using the script.
1 #!/usr/bin/env bash 2 # Anti-Throttling GET 3 4 file="$1" 5 shift 6 declare -i timeout=5 7 declare -i randbase=500 8 9 while getopts ":T:R:" opt 10 do 11 case $opt in 12 T) declare -i timeout=$OPTARG;; 13 R) declare -i randbase=$OPTARG;; 14 \?) echo "Invalid option: -$OPTARG" >&2;; 15 esac 16 done 17 18 while true 19 do 20 wget -c -T $timeout "$file" 21 declare -i r=$RANDOM%$randbase+1 22 echo "Waiting $r seconds..." 23 sleep $r 24 done 25 26 unset opt timeout randbase r
This is the first version of the shell script that I wrote. I made a modification to it which I will post later in this article, but I just want to focus a bit on the process I went through. This script is fairly succinct and easy to understand. It lets you set the timeout interval and the maximum time to wait. This maximum is used as the base for the modulus operator when it selects the random time interval. I’ve found that generally speaking the larger the file you’re downloading is, the longer you want to wait between download attempts.
I realized that waiting an integer number of seconds was still too obvious. What if the throttling bot is intelligent and can detect signatures of an automated process? I need to make the minimum increment small enough that the time it takes to process it will make the minimum waiting interval negligible. In other words, I want to make sure the time is not arbitrarily close to some interval, say 15.001 seconds, because an AI on the other end can easily see that that’s just an automated 15 second wait plus a small amount of time for processing. But if I make the intervals small enough, it won’t be able to separate the intervals from the processing time, which will make that aspect of automation undetectable. Here’s my final script:
1 #!/usr/bin/env bash 2 # Anti-Throttling GET 3 4 file="$1" 5 shift 6 declare -i timeout=5 7 declare -i randbase=500 8 9 while getopts ":T:R:" opt 10 do 11 case $opt in 12 T) let timeout=$OPTARG;; 13 R) let randbase=$OPTARG;; 14 \?) echo "Invalid option: -$OPTARG" >&2;; 15 esac 16 done 17 18 while true 19 do 20 wget -c -T $timeout "$file" 21 declare -i p=$randbase*100 # 1/100 seconds base 22 declare -i q=$RANDOM%$p+1 # 1/100 seconds delay 23 declare -i r=$q*10000 # microseconds delay 24 declare -i s=$q%100 # fractional part 25 declare -i t=$q/100 # integer part 26 echo "Waiting $t.$s seconds..." 27 usleep $r 28 unset p q r s t 29 done