To avoid these ads, REGISTER NOW!

Site Down

To avoid these ads, REGISTER NOW!

kbuhagiar

Well-known member
Joined
Dec 27, 2005
Messages
1,736
Location
Escondido, CA
It was a Cloudflare thing. The one site I manage also went down because of it.
Hope they have a permanent fix because Cloudflare was a godsend for us, it allowed us to keep our site up and running.
Without it we'd be history.
 

LXCam

ALLIANCE MEMBER
Joined
Apr 23, 2013
Messages
19,095
Location
AZ
All good Ryan, but it screwed up my morning coffee schedule something fierce 😆

The company I work for has become 1000% dependent on procore. Timecards, document processing, etc. many of us middle management folks have expressed our concerns over this migration.

Ya’d have thought the world ended this morning with this outage. What a shitshow.
 

txvwnut

Well-known member
Joined
Jan 1, 2015
Messages
7,603
Location
Bedford, Texas
Yeah just heard it on the radio that Cloudfare was down affecting people and businesses nation wide. Noticed it when I went to order some tires and my supplier link wouldn't work.
 
OP
R

Ryan

Administrator
Staff member
Joined
Jan 26, 2006
Messages
5,708
Location
Texas/Hawaii
That's probably the longest "almost global" outage I've ever seen? Started at around 4:30am central and only seems to be have fixed by 10:30am or so... My account rep says it was almost certainly nefarious.
 
To avoid these ads, REGISTER NOW!

Shiftless

Well-known member
Joined
Mar 9, 2014
Messages
14,500
Location
East Bay SFO
That's probably the longest "almost global" outage I've ever seen? Started at around 4:30am central and only seems to be have fixed by 10:30am or so... My account rep says it was almost certainly nefarious.
This outage once again showcases how vulnerable we are to a cyber attack. You don’t need a big army. You don’t need missiles.
Where is the redundancy to better protect business and banking systems?
 
OP
R

Ryan

Administrator
Staff member
Joined
Jan 26, 2006
Messages
5,708
Location
Texas/Hawaii
This outage once again showcases how vulnerable we are to a cyber attack. You don’t need a big army. You don’t need missiles.
Where is the redundancy to better protect business and banking systems?

Cloudlfare is sort of like a firewall. Them going down protects our data. But yes, I 100% agree.
 

ArcReactorKC

Well-known member
Joined
Jun 1, 2019
Messages
2,237
Location
Out in the county NE of KCMO
Every time a cloud system goes down, an on-prem server gets its wings.
yes and no. I have been a vehement supporter of on-prem and without our stuff being onsite we would have major issues.

But on the other hand all of our remote tools were down today because we use cloudflare as our DNS provider. It is literally the only "single point of failure" in the entire workflow.

I've got 5 power sources. Redundant supplies in every system and all systems have a clone on a rack in the shop with failover. 3 WAN sources.

And we still got bit by cloudflare, so it's time to figure out how I'm going to attack that. This was the first time we had this scenario but the attacks on cloudflare continue to grow.
 

Skellyii

ALLIANCE MEMBER
Joined
Nov 13, 2021
Messages
1,703
Location
KC Area
This outage once again showcases how vulnerable we are to a cyber attack. You don’t need a big army. You don’t need missiles.
Where is the redundancy to better protect business and banking systems?
There is some redundancy in place, but added redundancy costs money. Until businesses start losing money on a regular basis, don't hold your breath waiting for a bulletproof environment.
 

Ohio Andy

Well-known member
Joined
Jul 31, 2024
Messages
2,260
Location
Columbus, Ohio
Yeah, but you're not the only ones who had trouble because of this.... My employer sent out an emergency notification using the same system used for things like active shooter to let us know.
 

Skellyii

ALLIANCE MEMBER
Joined
Nov 13, 2021
Messages
1,703
Location
KC Area
Doesn't seem to be an attack this time:
November 18, 2025 at 9:48 AM

Cloudflare CTO speaks out:​

I won’t mince words: earlier today we failed our customers and the broader Internet when a problem in Cloudflare network impacted large amounts of traffic that rely on us. The sites, businesses, and organizations that rely on Cloudflare depend on us being available and I apologize for the impact that we caused. Transparency about what happened matters, and we plan to share a breakdown with more details in a few hours. In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack. That issue, impact it caused, and time to resolution is unacceptable. Work is already underway to make sure it does not happen again, but I know it caused real pain today. The trust our customers place in us is what we value the most and we are going to do what it takes to earn that back.

Sounds like a self-inflicted wound.
 

cody1325

Well-known member
Joined
Apr 17, 2024
Messages
1,078
Location
Southwest Virginia
It's weird how much of the web AWS and Cloudflare alone are responsible for.

That being said, most of the forums I'm on (all but a couple VBulletin-hosted) were down when I dragged my rear out of bed this morning).
 

CraigStu

Well-known member
Joined
May 22, 2014
Messages
4,017
Location
Blacksburg, Va
Yep, I ran out of forums to look into and had to get back to my bead board, chair rail project at 900AM instead of the more usual 930 to 10 am start of workday. 1st world problems like this ****.
 

NUTTSGT

Super Moderator
Staff member
Joined
Sep 14, 2009
Messages
50,884
Location
Northern Central Ohio
Doesn't seem to be an attack this time:
November 18, 2025 at 9:48 AM

Cloudflare CTO speaks out:​

I won’t mince words: earlier today we failed our customers and the broader Internet when a problem in Cloudflare network impacted large amounts of traffic that rely on us. The sites, businesses, and organizations that rely on Cloudflare depend on us being available and I apologize for the impact that we caused. Transparency about what happened matters, and we plan to share a breakdown with more details in a few hours. In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack. That issue, impact it caused, and time to resolution is unacceptable. Work is already underway to make sure it does not happen again, but I know it caused real pain today. The trust our customers place in us is what we value the most and we are going to do what it takes to earn that back.
Excuses are like, well. . . . we all know.

However, this one almost sounds like somebody that has a serious concern and skin in the game.
 

WillyBoy

Well-known member
Joined
Nov 10, 2021
Messages
635
Location
Genesee valley area of New York state
Other forums that I look at from time to time have been under attack lately by what the moderators call "bots".
They've gone to a captcha solve for log-in after the user name and password. Maybe connected, maybe not.
Multiple scams in the email lately, four times a day I get notified that my cloud storage is
FULL
BROKEN
NEEDS MORE MONEY
BLAH BLAH



sigh :(
 

ArcReactorKC

Well-known member
Joined
Jun 1, 2019
Messages
2,237
Location
Out in the county NE of KCMO
Other forums that I look at from time to time have been under attack lately by what the moderators call "bots".
They've gone to a captcha solve for log-in after the user name and password. Maybe connected, maybe not.
Multiple scams in the email lately, four times a day I get notified that my cloud storage is
FULL
BROKEN
NEEDS MORE MONEY
BLAH BLAH



sigh :(
Bot storms have become a daily occurrence to a lot of us. Usually in the form of DDoS

Our critical power website gets attacked daily, this has been going on for a few years now. Cloudflare has actually done a lot to help alleviate the issue, having quad redundancy for the critical systems applications has been the ultimate cure but I don't know for how much longer.

Looking at traffic on most of my domains it's less than 20% bots of any kind. Our critical power domain is over 90% I think it is attacked because on the surface is seems like critical infrastructure to a non-native English speaker.

As we do the ultimate relay from customer equipment to our app on their phones it's imperative that our systems work for customer ease of mind. (their equipment will do it's thing without our connection, we're just there so they can see the equipment remotely) This is one of the things that pushed me into my on-prem data center in the shipping containers. I'll need to make a thread about these when I get one of them to a respectable state.

Fortunately our app does direct IP resolve with no caching between so all of that works without cloudflare, and the failover does as well.

We're now migrating to a multi-provider authoritave DNS which means migrating away from cloudflare as a whole since they are a primary only and don't jive with having a secondary. Planning to use CloudDNS and Zonomi for this but I'll move one domain at a time to see how well it works. My concern is even if I move all of my domains to a more reliable system there is a jump point in the middle that is still cloudflare somewhere I may not know about.
 
To avoid these ads, REGISTER NOW!
Top Bottom