Hosting Down- Aug 15 9:30 EST

Jeffrey Hoffman · August 2017

Hosting is down. We access our database through the Vancouver data centre. Here is the note from the hosting status page on the BB website.

Event Notification 10163: We are experiencing an issue that may be affecting your Blackbaud Hosting application. During this event, you may be unable to access your site or database. We are tracking this event and will continue to provide you with regular updates regarding this issue.

https://www.blackbaud.com/profile/ahsstatus.aspx/

Jessica Heinze · August 2017

Jeffrey Hoffman:

Hosting is down. We access our database through the Vancouver data centre. Here is the note from the hosting status page on the BB website.

Event Notification 10163: We are experiencing an issue that may be affecting your Blackbaud Hosting application. During this event, you may be unable to access your site or database. We are tracking this event and will continue to provide you with regular updates regarding this issue.

https://www.blackbaud.com/profile/ahsstatus.aspx/

We are in the Boston Data Center and are experiencing issues as well. Yesterday it was telling us that the server could not connect and was experiencing issues and then that it was busy. Today it is telling us that no databases are found.

Ryan Hyde · August 2017

We're on the Boston data ceter and are having no issues so far today, though things really went haywire for a while yesterday afternoon. Got lots of error messages when doing normal things and then the system would crash (ODBC errors and autocorrect errors - if you've been hosted long enough you've seen these at some point).

Aaron Rothberg · August 2017

We're in the Boston data center and we also saw the No Database error this morning.

Jennifer Reager · August 2017

Boston Data Center here as well - I keep checking status.blackbaud.com for any updates, as advised by support yesterday. It has continually shown that all services are available, when they're not. I also haven't received any notifications from Blackbaud. Are there any other places I could get further information regarding the status of these technical difficulties? I'm just trying to figure out if I need to troubleshoot inside or outside our network.

Aaron Rothberg · August 2017

I'm getting a new error message this morning. I don't know if it's related to the issues in Boston, but it's definitely not normal. I created a constituent query with one criteria; Constituent Code is One Of Parent|Former Parent. I have three output fields; Constituent Id, First Name, Last Name. Nothing on the Sort tab. I run the query and get the error you see below. I closed the query, tried to click New Query, and I get the same error. I can't create new queries now and am in the process of relaunching RE to see if that resolves the issue.

f173ddc605f0e43df077e165816f0421-huge-od

Ryan Hyde · August 2017

Ugh. That's exactly the error we were all getting on Monday afternoon.

So far, I haven't experience dany issues (again, we also connect to the Boston dataceter). I'll update if I also see any problems. And I strong urge anyone who does experience these problems (or has a staff who experiences them) to open a case in support center. They won't be able to fix it immediately, but they need as much info as they can get typically.

Jen Claudy · August 2017

Aaron Rothberg:

I'm getting a new error message this morning. I don't know if it's related to the issues in Boston, but it's definitely not normal. I created a constituent query with one criteria; Constituent Code is One Of Parent|Former Parent. I have three output fields; Constituent Id, First Name, Last Name. Nothing on the Sort tab. I run the query and get the error you see below. I closed the query, tried to click New Query, and I get the same error. I can't create new queries now and am in the process of relaunching RE to see if that resolves the issue.

I've gotten this before...and not when experiencing other issues. Usually signing out and back in resolves the problem.

Aaron Rothberg · August 2017

I've just sent Blackbaud another email. I've had a case open for Boston data center latencies for over two months now. This morning I'm seeing latencies as high as 330 ms and those latencies are occuring at the same exact spot I find them each time, one hop just outside of ctx-login2.blackbaudhosting.com. This morning the problem node is das1-v3006.bo3.savvis.net.

b4dd43a084ed7715f64dd2695bdfd8bd-huge-la

Alison Hall · August 2017

We've been having issues with Raiser's Edge since early last week -- mostly ODBC errors. Is this only happening with the Boston Data Center? Why aren't we getting any notifications from Blackbaud about the ongoing situation? Am I missing something? I always check Hosting Status but there is rarely anything reported besides maintenance. Today there was this... but no description about exactly what the issue was. Is there a way to look up what 10166 and 10165 were about? Event Notification 10166: This issue has been resolved and services have now been restored.

Event Notification 10165: The issue has been resolved and services have now been restored.

Ryan Hyde · August 2017

I don't know why this isn't being reflected on the hosting status page - from the comments here and elsewhere, it does seem to be hitting just Boston (no surprise really there - Boston seems to have the most trouble of any of the datacenters).

I just got my first ODBC error of the day a few minutes ago. Sigh.

roger berg · August 2017

Hosted through Boston - had just one of the odd ODBC messages yesterday - logged out and back in and that took care of it.

Karen Stuhlfeier · August 2017

Yes, logging off and signing back in seems to work - but it's really a pain when you're in the middle of working on something.

Aaron Rothberg · August 2017

They finally acknowledged the issue. I don't know if anyone else contacted them, but I emailed the person that owns my two-month old boston latency case and he contacted hosted engineering. It appears it's also affecting Education Edge and Financial Edge.

b0b4f41beaed089d18cd7c7360d03629-huge-ho

Jessica Heinze · August 2017

We keep getting a runtime error "o", can't complete batches, I got an error messasge of "Can not call friend function on object which is not an instance of defining class", and an Automation error "The callee 9server[not server application]) is not available and disappeared; all connections are invalid. The call may have executed" then followed by another run-time error!

Ryan Hyde · August 2017

Yes, that's another of the common errors you get when something breaks on the server you're connected to (though I haven't personally seen that one recently). Most of the time, you can force close RE and log back in and go about your business, but sometimes the error persists even after you log out and back in again. I would open a case with Hosting to let them know what you're experiencing.

Joan Perry · August 2017

We are hosted out of Boston and have been having similar issues...especially this week. Staff have been unable to login in or have been kicked out of RE. I have received messages once I log in that I can access the different modules, problems logged into the NetCommunity plugin and getting errors trying to open transactions.

Looking at the hosting site, it appears there were event notifications that were resolved and then new ones arising. Wishing there was better communication about what is happening and/or who to contact. It's hard to remember to take screen shots of error messages all the time when you are trying to get work done...I'm too often kicked out and try to re-login before I think about the screen shot...let alone getting other staff to do all of that.

Jennifer Reager · August 2017

Jessica Heinze:

We keep getting a runtime error "o", can't complete batches, I got an error messasge of "Can not call friend function on object which is not an instance of defining class", and an Automation error "The callee 9server[not server application]) is not available and disappeared; all connections are invalid. The call may have executed" then followed by another run-time error!

I get this one every single day. . .pretty much since we've been hosted (few years now).

I had a relatively quiet day yesterday after Monday afternoon's stuff, but my RE is really going haywire today. It's always fun when you get error messages and things that you've never seen before!

Ryan Hyde · August 2017

Yes! Fun! That's exactly it :P Anything to shake up the day, haha

Nicole Holt · August 2017

Jen Claudy:

Aaron Rothberg:

I'm getting a new error message this morning. I don't know if it's related to the issues in Boston, but it's definitely not normal. I created a constituent query with one criteria; Constituent Code is One Of Parent|Former Parent. I have three output fields; Constituent Id, First Name, Last Name. Nothing on the Sort tab. I run the query and get the error you see below. I closed the query, tried to click New Query, and I get the same error. I can't create new queries now and am in the process of relaunching RE to see if that resolves the issue.

I've gotten this before...and not when experiencing other issues. Usually signing out and back in resolves the problem.

I've logged out/back in at least a half dozen times this morning. And sent an advisory to staff to save often. So signing out/in does resolve it, but only briefly. There is clearly a bigger problem at hand.

Ashley Holm · August 2017

Nothing new to add... just acknowledging that misery loves company. It's bonkers today!

Ryan Hyde · August 2017

One of my users just sent me this. I still see an ODBC error in there, but presented a whole different way.

f3c625faca51b85bc7760d8fc90c6d07-huge-im

Joan Perry · August 2017

This is one of the error screens I got today...plus the one asking me if I wanted to delete transactions in the NetCommunity plug...

Oh the fun we are having!

Aaron Rothberg · August 2017

...and it's getting worse. Now it's affecting Altru and CRM.

0de9c44792e7fa2e129b9337973d6cb1-huge-ho

Rebecca Altes 2 · August 2017

My new one is Abnormal termination of SuperGrid70Ex. I'm also consistently getting the joyful No Databases Found.

Jennifer Reager · August 2017

**twiddles thumbs**

So, anyone heard any good jokes lately?

Aaron Rothberg · August 2017

So a piece of string walks into a bar and says, "Bartender, gimme a drink" and the bartender says, "Sorry, we don't serve pieces of string here." So the piece of string walks outside, ties a knot in the middle of himself and frayes his ends a bit. He walks back into the bar and says, "Bartender, gimme a drink." The bartender says, "Hey, wait a minute. Aren't you that same piece of string that was in here a minute ago?" and the string says, "Nope. I'm a frayed knot".

Suzie Capps · August 2017

I can never miss the opportunity for a good stats pun...

Jessica Heinze · August 2017

Ryan Hyde · August 2017

Here's one of my favorite XKCDs in a while:

1ff04264c788af61966615125e0bbaea-huge-xk

source: https://xkcd.com/1839/

Holly Herbert · August 2017

Hi everyone,

I am so sorry you're experiencing latency issues. Rest assured that I have been actively listening to this discussion thread and have been diligently escalating everything you share here with appropriate leadership within Blackbaud. In the meantime, we continue to push updates to our hosting status page. As I learn more, I will continue to update this thread. If you continue to experience issues that are not reflected on the hosting status page, please post them here and I will continue to update leadership with your feedback.

Thank you,

Holly

Community Manager

Hosting Down- Aug 15 9:30 EST

Comments

Categories