James Turner (jturner@shield-legal.com)
2025-06-12 13:46:25

@deleted-U06C7A8PVLJ @Ryan @Nicholas McFadden @Dustin Surwill Did anyone change any permissions or run anything in Bigquery today? Out of nowhere the majority of our scheduled queries started failing. It's happening to pipelines that don't necessarily go together.

image.png
Nicholas McFadden (nickmcfadden@shield-legal.com)
2025-06-12 13:46:55

I redoplyed the Leadspedia info

Nicholas McFadden (nickmcfadden@shield-legal.com)
2025-06-12 13:47:03

but that is it

Dustin Surwill (dsurwill@shield-legal.com)
2025-06-12 13:47:31

I did not do anything. I am also seeing the errors

Nicholas McFadden (nickmcfadden@shield-legal.com)
2025-06-12 13:47:47

I am seeing that central1 stuff is down

deleted-U06C7A8PVLJ
2025-06-12 13:47:47

Nope nothing from my end

Brian Hirst (brian@themedialab.agency)
2025-06-12 13:48:18

I’m getting errors. I can’t look at some tables

Dustin Surwill (dsurwill@shield-legal.com)
2025-06-12 13:48:28

https://status.cloud.google.com/

Dustin Surwill (dsurwill@shield-legal.com)
2025-06-12 13:48:36

Lots of pieces of google down atm

Nicholas McFadden (nickmcfadden@shield-legal.com)
2025-06-12 13:49:05

I was just about to do the same

Nicholas McFadden (nickmcfadden@shield-legal.com)
2025-06-12 13:49:27

looks like it is down

Nicholas McFadden (nickmcfadden@shield-legal.com)
2025-06-12 13:49:47
image.png
deleted-U06C7A8PVLJ
2025-06-12 13:50:19

It did say to me I needed permissions to go into cloud scheduler which was weird since I could go on there in the past

Dustin Surwill (dsurwill@shield-legal.com)
2025-06-12 13:52:07

*Thread Reply:* That is probably related to the identity service being down

deleted-U06C7A8PVLJ
2025-06-12 13:52:30

*Thread Reply:* Ohhh okay. That makes sense. Ty 🙏

Joe Santana (jsantana@shield-legal.com)
2025-06-12 13:50:37

@Joe Santana has joined the conversation

Dustin Surwill (dsurwill@shield-legal.com)
2025-06-12 13:50:49
image.png
👍:skin_tone_3: Zekarias Haile
James Turner (jturner@shield-legal.com)
2025-06-12 13:52:14

Well good to know it's not something we caused but dang- It's not even just the Americas. It's every region around the world. Cloud Redundancy be damned.

🔥 Dustin Surwill, Zekarias Haile, Brian Hirst, Ryan
James Turner (jturner@shield-legal.com)
2025-06-12 13:53:22

https://status.cloud.google.com/index.html

Ryan (ryan@themedialab.agency)
2025-06-12 13:55:38

Whoops, geez Google

James Turner (jturner@shield-legal.com)
2025-06-12 13:59:54

Maybe not just google?

https://downdetector.com

downdetector.com
Status overview
Realtime overview of issues and outages with all kinds of services. Having issues? We help you find out what is wrong.
Original URL: https://downdetector.com
Dustin Surwill (dsurwill@shield-legal.com)
2025-06-12 14:01:12

Well, the google, aws and cloudflare outages could be partially related and cause the rest of them

Joe Santana (jsantana@shield-legal.com)
2025-06-12 14:02:06

Cloudflare is having issues

Joe Santana (jsantana@shield-legal.com)
2025-06-12 14:04:12

https://www.cloudflarestatus.com/

cloudflarestatus.com
Cloudflare Status
Welcome to Cloudflare's home for real-time and historical data on system performance.
Original URL: https://www.cloudflarestatus.com/
Dustin Surwill (dsurwill@shield-legal.com)
2025-06-12 14:10:04

Looks like our Postgres DB in Google Cloud SQL is affected, integrations have started throwing connection errors

James Turner (jturner@shield-legal.com)
2025-06-12 14:12:23

It's expanding

image.png
Joe Santana (jsantana@shield-legal.com)
2025-06-12 14:24:51

looks like they identified the issue and are recovering

Dustin Surwill (dsurwill@shield-legal.com)
2025-06-12 14:25:14

Cloudflare has but not Google yet, https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW

Dustin Surwill (dsurwill@shield-legal.com)
2025-06-12 14:33:00

We will have to wait for us-central1 for a good chunk of our automations

image.png
James Turner (jturner@shield-legal.com)
2025-06-12 15:11:37

Dang

image.png
James Turner (jturner@shield-legal.com)
2025-06-12 15:38:21
image.png
Joe Santana (jsantana@shield-legal.com)
2025-06-12 16:49:54

I got some input from other google centric sources that there seems to be at least, some recovery, how are we doing?

James Turner (jturner@shield-legal.com)
2025-06-12 16:52:04

We can now see majority of our data, which we Could not before, but scheduled transformation is still hit and miss.

A lot of our main transformation pipelines run on Central1 which was the last to be fixed.

James Turner (jturner@shield-legal.com)
2025-06-12 16:52:19

That was last I checked, I'm going to try again shortly.

James Turner (jturner@shield-legal.com)
2025-06-12 16:54:35

Looks like scheduled queries are back up!

James Turner (jturner@shield-legal.com)
2025-06-12 16:54:54

We can now see majority of our data, which we Could not before, but scheduled transformation is still hit and miss.

A lot of our main transformation pipelines run on Central1 which was the last to be fixed.

James Turner (jturner@shield-legal.com)
2025-06-12 16:56:22

Identity and access management is still down for all of the Americas, but as long as we stay logged in and don't need to give/remove access so far so good on the GCP side.

James Turner (jturner@shield-legal.com)
2025-06-12 17:01:20

Looks like Cloud Services are running again

👍 Joe Santana
Joe Santana (jsantana@shield-legal.com)
2025-06-12 17:06:22

Do we have the issue publicly released yet?