I agree that OP sounds like a beginner, and what you’ve suggested is likely the best approach for someone who is familiar with frontend tools and frameworks. Selenium (and admittedly BeautifulSoup) is probably too low level for this particular user, but that doesn’t mean they can’t still learn some fundamentals while solving this problem without resorting to something as heavy and complicated as background browser emulation and rendering. I could be wrong though.
I’m not currently on Discord, could you upload the code to pastebin or something similar?
I would love to see your code, but I understand if this forum isn’t the most ideal place to share.
In my experience, this scenario typically means that there is some sort of API (very likely undocumented) that is being used on the backend. That requires a bit more investigation and testing with browser developer tools, the JS Console, and often trial and error. But once you overcome that (admittedly very complex and technical) hurdle, you can almost always get away with just using the requests library at that point.
I’ve had to do that kind of thing more times than I’d like to admit, but the juice is almost always worth the squeeze.
Selenium is really more of a testing framework for frontend developers, and could theoretically be used for scraping, but that would be somewhat like buying a car based on the paint and not looking in detail under the hood.
I can’t say I’ve ever worked with scrappy, but the tool I would use for web scraping with Python is BeautifulSoup. This tutorial seems decent enough, but you will need to understand basic web concepts like IDs, classes, tags, and tag attributes to get the most out of the tutorial: https://geekpython.medium.com/web-scraping-in-python-using-beautifulsoup-3207c038723b
W3Schools will also be your friend if you have questions about HTML/CSS selectors in general: https://www.w3schools.com/html/default.asp
Understanding regular expressions and/or xpath would also be very helpful, but are probably best considered to be extra credit in most cases.
I’ll try to respond if you have any issues or questions, but hopefully that gives you enough to get started.
Oh yeah that’s a great alternate option too if your mobile plan includes tethering. I’ve successfully used both Android and iOS tethering in the past and it was pretty seemless each time.
Wi-Fi drivers are notoriously complicated on Linux in general, though things have been improving. But yeah if ‘iwctl device list’ comes up empty when you plan to use Wi-Fi to install Arch, especially if Ethernet isn’t a viable temporary alternative because your device doesn’t have an Ethernet port, you’re in for a tough time.
Those aren’t really the same apps between the two systems, even if most of them still have the same names. Not only do the modem versions have more features, but they also have updated designs, window chrome, and use more secure underpinnings.
Compare the boot times, network speeds, ease and speed of installing new software, and security of the same old vs. new systems, and the new stuff will come out on top every time.
See my other reply here for a breakdown of what it means to break userspace. One recent example was when Apple removed support for all 32-bit applications in macOS 10.15 Catalina. It’s something they do quite regularly with the attitude that app developers can either update their apps, or their apps will simply not run on macOS going forward.
It’s not necessarily a bad thing to force developers to update their apps in this way, but it does mean that macOS does not have backwards compatibility at nearly the same level as Windows or the Linux kernel. If you care about running older software (say as a business with a critical application that would be too expensive to replace/update, or to play an old game on your modern machine), macOS is likely a non-starter.
100% this. Every website is different, though after doing this kind of thing for long enough, there are often common patterns and frameworks/libraries. Even general obfuscation can be reasonably reverse engineered with enough time and effort.