The only scraping I do currently is a once daily scrape of every achievement to update the ratios.
We have a specialised API call where I can pass in a gamertag and an achievement id, and the response is just a simple “yes/no”.
The scans will naturally get quicker as the month progresses because there’s no need to query the API for achievements we already know are unlocked.
However, the API only accepts one achievement at a time, per query, which is what makes it slow. If I could pass in multiple achievements this would speed things up considerably, but again, it would need a change at TA’s end.
I don’t think we’d gain much in the way of performance by hitting everyone’s gamer page and parsing the HTML, actually I think it would perform worse ultimately. Hitting gamer’s pages on TA is also a lot heavier on TA than the simple API call I hit, so if I started hammering them it’s more likely to actually impact TA (not that this is a huge concern, they have big servers).