全部文章
10 分钟阅读glizzykingdreko

Training a captcha solver by reading the generator instead of the images

Ticketmaster PH and SG gate add-to-cart with a 4 letter captcha. It comes from Yii, which is open source, so I ported the generator and trained a small ONNX model on captchas I render myself.

Training a captcha solver by reading the generator instead of the images

Ticketmaster PH (Philippines), exactly like Ticketmaster SG (Singapore), puts a 4 letter image captcha on some actions, like adding a ticket to cart. Instead of throwing a generic OCR at it, I ported the captcha generator to Python and trained a small model on captchas I render locally.

Where it shows up

After picking a ticket, the ATC lands you on /ticket/check-captcha/.../.../.../... and you get the captcha. 4 letters, white on blue, 120x100.

A real captcha from the ATC page

The Oracle

When working in this kind of situations, for training an ML model for a specific type of challenge, you obv need a way to verify that your current output is correct, therefore you need an oracle.

The website

In this situation the most logic, but as well slower and more expensive in terms of time and money solution would have been making a parallel script that enters the website, initializes the session and solves the antibots, adds a product to cart, and validates a captcha each time.

fresh session proxy, antibot solved add to cart reach the captcha read it submit a guess the site answers right or wrong every label costs a proxy usage and time 1 label, then the whole run again

Even if this is the safest and most reliable solution most of the times, in this case it would have took extremely lot of time and resources.

A vision model

Another option would have been pulling a big quantity of captchas from the website by using

"https://ticketmaster.ph/ticket/captcha?v=1" # On each refresh a new captcha is returned

And then use an external oracle such as a vision model or even human classification for validating each output of our model.

vhkq pull a captcha /ticket/captcha a vision model or a human reads it vhkq ✓ vhkg ✗ no label yet costs a call, or your time and error labeling are possible

A way slower and maybe even more expensive.

So, I started checking the WAF page source.

It's an open source Captcha

Let's take a look at the script that loads it:

Yii script tag

https://ticketmaster.ph/assets/bba9a1aa/yii.captcha.js

It presents itself as the "Yii Captcha widget".

Yii script header

By searching it on google, comes out that the captcha is Yii2, and Yii2 is open source, which means I can read the function that draws it and port it to a local script: CaptchaAction::renderImageByGD.

$x = 10;
$y = round($this->height * 27 / 40);
for ($i = 0; $i < $length; ++$i) {
    $fontSize = (int) (rand(26, 32) * $scale * 0.8);
    $angle = rand(-10, 10);
    $letter = $code[$i];
    $box = imagettftext($image, $fontSize, $angle, $x, $y, $foreColor, $this->fontFile, $letter);
    $x = $box[2] + $this->offset;
}

Per letter: a random size between 26 and 32, a random tilt between -10 and 10 degrees, then move right.

Yii source code

The font is SpicyRice. Stock Yii draws 6 to 7 letters from bcdfghjklmnpqrstvwxyz and mixes vowels in. The captchas I sampled on Ticketmaster were 4 consonants, so the generator uses bcdfghjklmnpqrstvwyz, which gives 160,000 possible codes.

Comparing Yii defaults with the target, they changed 5 things (on the TM implementation, guessed by analyzing some loaded captchas):

Setting Yii default Ticketmaster PH / SG
Image size 120x50 120x100
Background white #FFFFFF #00467f
Letters color #2040A0 white
Code length 6 to 7 4
Letters consonants and vowels consonants only

Porting it

From there it is a direct translation into Pillow:

def render(code, random):
    # CaptchaAction::renderImageByGD ported from PHP
    image = Image.new("RGB", (WIDTH, HEIGHT), BACK_COLOR)
    ...
    for letter in code:
        point_size = int(random.randint(26, 32) * scale * 0.8)
        angle = random.randint(-10, 10)
        # each glyph is drawn on its own layer so it can spin around the baseline origin
        layer = Image.new("L", (pad * 2, pad * 2), 0)
        ImageDraw.Draw(layer).text((pad, pad), letter, font=load_font(point_size), fill=255, anchor="ls")
        if angle:
            layer = layer.rotate(angle, resample=Image.BICUBIC, center=(pad, pad))
        image.paste(FORE_COLOR, (x - pad, y - pad), layer)
        # gd advances to the x of the rotated lower right corner, which is not the widest one
        left, top, right, bottom = glyph_box(point_size, letter)
        radian = radians(angle)
        x = x + int(round(right * cos(radian) + bottom * sin(radian))) + OFFSET
    return image

python generator.py opens a random one in your image viewer, and sample() gives you (code, image) pairs from a seed.

Six of them, created locally:

Captcha rendered by the Python port Captcha rendered by the Python port Captcha rendered by the Python port Captcha rendered by the Python port Captcha rendered by the Python port Captcha rendered by the Python port

Training on captchas I render myself

Now there is no dataset to collect. Every training step renders a fresh batch in memory and the labels are the codes I asked for, so no proxies, no requests to the site, no manual labelling.

dzzy pick a code render it the model dzzy dzzy the answer its guess same 4 letters? if not, correct the model and go again

Pillow and GD do not draw exactly the same pixels, so the training images get a light blur, a brightness jitter and some noise on top. That keeps the model from learning edges that only Pillow produces. Validation runs on 3000 captchas from a different seed than the training ones.

The model is a small CNN reading the image as 4 slots of 20 letters. Open trainer.py, set RELEASE, run it:

python trainer.py micro
[3] Training for 300 steps
    step   100/300  loss 2.4031  full  15.87%  per letter  63.98%     5s
    step   200/300  loss 0.3797  full  89.83%  per letter  96.82%     8s
    step   300/300  loss 0.1542  full  94.07%  per letter  98.12%    11s

full is the share of captchas with all 4 letters right, the number that matters. per letter just shows how far off the misses are. That run took 11 seconds. The default 4000 steps take a few minutes on an M4 Max.

You get 2 files: a .pt to resume training from, and a .onnx the solver uses, so production needs ONNX Runtime and not PyTorch.

The models

3 sizes, same structure, the last layers are wider each time:

Release Parameters ONNX size Accuracy Speed per solve
pico 45,488 186 KB 99.19% 0.60ms
nano 59,744 243 KB 99.49% 0.61ms
micro 139,808 563 KB 99.87% 0.65ms

Accuracy is full codes on 20,000 fresh captchas from benchmark.py, speed is the median of 900 solves on 1 CPU thread of an M4 Max. Those captchas come from my port of the generator, not from Ticketmaster itself, so real ones may score a bit differently.

So why 3 models you may ask... can't a guy just have fun?

micro reading fresh ones, with the per letter confidence next to each:

Solved captcha with expected code, confidence and solve time Solved captcha with expected code, confidence and solve time Solved captcha with expected code, confidence and solve time Solved captcha with expected code, confidence and solve time Solved captcha with expected code, confidence and solve time Solved captcha with expected code, confidence and solve time

Conclusion

This has been a quick and easy case study for a different approach of a ML model trianing, saved me lots of time and resources.

Repo is here with mnodels, trianing script and all the other scripts used.

Need antibot solutions?

TakionAPI, we handle the wall. You ship the rest.

If you would rather not deal with captchas and antibots yourself, TakionAPI covers DataDome, Akamai, PerimeterX, Cloudflare, hCaptcha and the rest, one HTTPS call per token, no browser. We know what we are doing. Start with a trial.

We offer as well cheap and reilable tmpt solutions for Ticketmaster.

Connect with me

If you enjoyed this, follow me on GitHub and Medium to get a ping whenever I post or open-source something.

Sources

captchaticketmasteryiimachine-learningonnxocrreverse-engineeringticketing

跳过逆向工程。

Takion 为每一道主流反机器人防护墙返回新鲜的 cookie、请求头和令牌。一次 POST,无需浏览器,一小时内完成首次调用。