Yea, it's like we have a super smart alien computer that can do anything but we need to spend massive amounts of time building fences and moats and write extremely detailed instructions so it doesn't run away and try to "help".
I definitely fall into that, on linux it's just a lot of extensions, scripts etc. Thing is, it's also brittle and not really useful for anybody to talk about it? Not sure I get what you mean with the third sentence.
My naive take on a Cloudflare perspective wants to combine "three times is enemy action" with toddler-speed block dropping and manual clearance. What's the money reason this problem isn't handled at the ISP level?
I don't quite understand your post, but is your question why don't the ISPs of the sources of the abusive traffic sort it out?
The distributed nature of DDoS means each participating host isn't sending that much traffic, and there are often tens or hundreds of thousands of participating hosts. An ISP should verify claims of abuse before cutting off customers, and since most of the customers are presumably unaware of what their systems are doing, there will be a lot of unhappy customers and then you've got to spend a lot of customer support time on helping them clean up their systems so they can get back online.
I spent a fair amount of time sending out abuse reports for phishing / malware senders about a decade ago, and most abuse reporting addresses are a black hole. Even if you do get to someone who will do something about abuse, they won't do it quickly.
There's be a few high profile longer term DDoS attacks lately, but when I was running infra that got a lot of stuff, it was mostly people kicking the tires on DDoS as a service offerings and most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that.
Starting from there, this DDoS sounds like layer 7 DDoS which is easy to track to the immediate senders, but a ton of DDoS is volumetric stuff, often volumetric reflection attacks where the senders spoof your address. If you're getting that, best you can do is get the reflectors kicked off (or cleaned up) ... tracing back to the sending hosts means getting a reflector (and their ISPs) engaged to do a lot of labor intensive work.
All of that investigation stuff takes qualified people lots of time, that's your money reason it doesn't happen.
> most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that.
Every victim (such as readthedocs), or even people sharing blocklists to avoid becoming a victim, blocking that ISP's ranges until they do clean up their network could be a convincing argument?
As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. Such a reoccurring source of abuse needs to either clean up or find themselves a different ISP to spread harm onto the net
I get what you're saying about that this won't solve an ongoing attack right this minute, or even by next week. But if we just let it all happen then the solution is going to be either (1) we all buy equipment that can handle something like a terabit per second and arm our infrastructure to the teeth or (2) centralize all traffic through a vetting entity who decides which client gets to visit the internet today. So far we're headed towards the latter and nobody really wants that. Abuse messages will have to slowly trickle down from victims to originating ISPs to users, and if users didn't willingly sign up, then to wherever users are getting this malware (Google's app store will be a big component). Stopping this at the source seems to me a much more desirable long-term solution
> As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious.
It sure is --- but an ISP would want to observe the traffic themselves, and if it's a 90 second attack every so often, chances are they won't see it when they look. When it's volumetric reflection, you can probably tell them how to send a request and see the response, and maybe they'll contact the customer, but maybe they'll just sit on it. As a victim, the ROI for reporting just wasn't there.
I wasn't getting huge traffic flows, and I was mostly getting attacks against www, which wasn't my actual service, so making sure volumetric attacks below my interface rate were shrugged off and taking simple actions like dropping requests from http clients with user-agent Wordpress were good enough. If the volumetric attacks were much over 10G, my host would have null routed my servers, which is annoying but highly scalable --- many ISPs support a BGP blackhole community, so my host can add my attacked IP to that and their upstreams will drop inbound packets when they enter the ISPs network.
I can't find a reference now, but I've seen things that allowed for more specific blackholing, such as by source or destination port number or by protocol. If my host's ISPs are dropping all UDP and IP fragments to my IP under attack, I could keep serving my TCP traffic and ignore a huge DDoS. I wouldn't even be able to measure the size of the DDoS.
> an ISP would want to observe the traffic themselves
My ISP didn't, when they got access logs from a service I attacked as a teenager and asked me to explain that to get the connection unblocked
Idk, at the moment we're simply not even trying to set a standard. Maybe it would work reasonably well when the ISP needs to observe the traffic and, after a few days of the initial report, they observe a netflow that matches a new abuse report. Even if we set low standards, currently, too few people are sending abuse notifications instead of just sticking it behind the great internet vetting service and calling it good
> maybe they'll contact the customer, but maybe they'll just sit on it.
That's the core point no? If they don't care about their abusive traffic, nullroute their ranges. If admins consistently do that, the abuse has to stop or the ISP goes out of business
www.uceprotect.net does that for email. It's a DNS blacklist that puts hole networks on it, even if "only" a individual hosts SPAMs.
It's a double-edged sword. That is how you end up with most ISPs blocking port 25 completely. If your hosting provider is on the list you are collateral damage. You yourself can do very little to remedy the situation except to beg your provider "to look into it".
What do you expect the ISPs to do in this story? We are talking about TSL connections. Block port 80 and 443 and expect the costumers to use your HTTPS-Proxy. Than they could inspect and block individual actions.
> If your hosting provider is on the list you are collateral damage.
Yes, and this sucks. I moved ISPs because the original one had burned IP addresses that you can't send email from. Every ISP that gets the ranges burned like that will eventually either goes out of business or gets their act together. Not by tomorrow, but eventually
Like, the only other alternative outcome I see is that everyone has to pass through a central surveillance point that decides who's benign and who's naughty, and since nobody wants that... what else are we to do but report abuse?
> What do you expect the ISPs to do in this story? We are talking about TLS connections.
What they've done to me when I abused a service as a teenager, the ISP got an abuse notification: cut off the connection, ask the subscriber wtf this traffic is and how they're going to make sure it doesn't happen again (at which point my dad, the subscriber, came to me and asked if I knew something about this.... yeah ^^')
TLS doesn't matter because the ones receiving the abusive traffic can say what it was. Their access logs will contain the decrypted information and that should be put in an abuse report. If the customer denies everything, turn on netflow logging for a month and see if a future abuse report comes in that can be correlated against the netflow logs. Or have netflow logs stored for 1h by default and retain the entries for which an abuse notification came in (grep for IP addresses in incoming notifications). Lots of reasonable options there; this isn't the difficulty. It's getting people to send that abuse notification so the ISP can identify the problematic subscribers
Do you think personal responsibility travels uphill or downhill?
Each of the letters in CEO stand for a word with specific meaning, and when you put them together they tell you where the buck stops. It’s not a job for people who can’t take responsibility for their fuckups.
> Do you think personal responsibility travels uphill or downhill?
This is trying to be pithy but it’s not. Every leader from the CEO down has responsibility for their area of control and influence.
> It’s not a job for people who can’t take responsibility for their fuckups.
Neither is EVP or any other title that ends in “VP”.
The CEO ultimately owned The fact that installing movie maker was a shit show, but so did everyone in a position of leadership that contributed to that. Accountability does not mean everyone gets to point at the CEO and say “ultimately it’s their fault” as if that absolves them.
This says more about the electorate than the candidates. You can’t have a segment of your population that refuses to connect ideas and expect a working society to come out the other end.
they connect ideas all day. xenophobia is that, making fast and loose connections.
unfortunately we are deep into the TikTok times, and simple, but catchy ideas dominate. it's flat Earth, QAnon, anti-vax, anti-economics but now in 4K.
How so? As long as it works to spec, I haven't had anyone care. They literally hire people so they don't need to care about the details. Put money in, get working software out.
And, AI is rapidly getting better than people at both code review and authorship, so a human deeply involved is turning into nothing but a slowdown. The main purpose people have is testing that the specs were, in fact, implemented properly.
I guarantee your specs/testing are either inadequate and/or you're not leveraging lots of existing (and probably free open source) code that was already written by humans and meets the spec better without ever needing an LLM.
The vast majority of properly written software was already plumbing well over a decade ago. The software engineering is making high level decisions based on experience with respect to the existing tools and the needs of the business. If you're not already using LLMs that way, you would have been a similarly bad manager of human devs writing similar inadequate slop. Less code has always been better code.
The line in the sand for these arguments really ought to be whether you think LLMs are better than humans who actually know what they're doing.
If you think LLMs are better, or could get better while continuing to use statistical methods, you automatically lose the argument (delusional/ignorant) and any hope of regaining credibility. That's not dogma. That's the science.
reply