Yet more Queue Problems in the Azure Hosting Environment

Options
Once again, we're experiencing unacceptable quality issues with Queue running in the Azure hosting environment.  We reached out to Blackbaud support last week but they are still "evaluating" the situation and not moving particularly fast on this.


Here are a few of our issues:

Delayed Running Queues: if we set up a Queue to run Tuesday night at 8pm it won't actually run until Wednesday around 7:55pm.  Our Queues seem to consistently be running like this.  

Locked Queues: We have a a Queue task that's essentially "locked" by Queue server and won't run and won't allow us to edit or cancel the schedule.  It just sits there.  When we try to work on it we get an error that says it is locked by a machine, but that machine's address is the server itself.  

Emailed PDF Files on a Scheduled Queue are Corrupt:  We can save PDF files via a scheduled Queue, but when we have Queue send an email with the PDF file the PDF file is sent... but it's corrupted (actually the file doesn't seem to be valid or big enough to be a real PDF file).


All of these issues have been reported to Blackbaud support and the only answer we've received is "eh, we're looking into it."   Again, we paid extra for Queue and again, we feel as though we were sort of cheated. 


Is anyone else having these issues with Queue?  Is anyone else even USING Queue in the RE Hosted environment?

 
«1

Comments

  • Good morning Tom,


    I didn't realize Queue was a paid add-on.  It must have been included with the total price when we converted.

    Anyhow, it has never worked correctly for us, from day 1 of conversion.  Consultant and developer worked on a couple for us probably a year ago, and they never got it working.

    I just glanced at one I setup and tested exactly two years ago today, and it completed successfully on the setup, but never completed the next run date.


    Of course, we have been hosted since conversion, so I don't know what it was like pre-hosted.  Were you able to actually email others using an email program, or did you email yourself only?


    Larry
  • Hi, Larry!


    Yeah, we paid a good bit extra for Queue when we converted to Blackbaud hosting and RE NXT back in November of 2018 and we've had 11 months of problems and run-around from support.


    We were told we could email reports directly out of RE Database View (hosted) but our experience with that has been dismal at best.  Instead we settled on the idea of having Queue simply save out all the reports and files we wanted to a shared directory and then use another process to pick up the files and email them out on a set schedule.


    We had it working like that for about 9 years when we had the Blackbaud server onsite.  Now that we're hosted we lost all those automation capabilities.  We have been able to eliminate the need for a few of those reports with Insight Designer, but a lot of the nightly "file dumps" are now done weekly, if at all.


    Any reports we send out view email are done manually at the moment because we simply can't get Queue working in any reliable way.
  • Tom Klimchak:

    Hi, Larry!

    ...

    We were told we could email reports directly out of RE Database View (hosted) but our experience with that has been dismal at best.  Instead we settled on the idea of having Queue simply save out all the reports and files we wanted to a shared directory and then use another process to pick up the files and email them out on a set schedule.


    We had it working like that for about 9 years when we had the Blackbaud server onsite.  Now that we're hosted we lost all those automation capabilities.  We have been able to eliminate the need for a few of those reports with Insight Designer, but a lot of the nightly "file dumps" are now done weekly, if at all.


    Any reports we send out view email are done manually at the moment because we simply can't get Queue working in any reliable way.

    We were told the exact same thing about emailing reports.  It seems now that capability will only be fully implemented with NXT, or should I say the extra cost of Reeher.

     

  • We're running Queue on Azure hosting, and it is working fine, however we are not sending files through email - only job notifications. 


    We're a little constrained by only having the ability to export to the hosted directory, but for large overnight exports for reporting, I'm just using FileZilla to transfer the source report data from hosted to our networks - I'd like an automated tool, but have not found one as yet. Our closest hope was Dell's Boomi, however the 90 password forced resets closed that avenue off for us (internal politics I think)


     
  • Thanks for the add'l info, Tom Klimchak. I wonder if all this is signalling an eventual pull back from of Queue delivery of report results in the form of PDFs attached to e-mails. Might this coincide with some underlying desire to move away from Crystal reports (as the basis for canned reports in RE) and toward SKY reporting or something? I hope not, especially if it means we'll be told we have to purchase some other add-on in order to auto-deliver reports via email. We're already paying a lot for RE hosting/NXT. Unfortunately, I've not found another way in RE to pull the specific types of totals needed and share them automatically.


    Update: Shortly after posting my original reply above, I did hear from the support representative. He asked me to run a test and report to them on the results, which i did on Dec 31. Unfortunately, the test results seemed to provide no help for further diagnosis and I've not heard back yet.
  • I retract my previous post.


    It was working fine, until a random update was applied, and since early November, it has been nothing but grief. 

    I'm seeing things like automated data integrity queries refreshes randomly failing owing to timeouts, or queues failing owing to "Failed to load control ‘ctl_LockOutScreen", or even out of memory issues. Support are bouncing back to developers, who are silent. 


     
  • I can relate to Rob Wild's "Support are bouncing back to developers, who are silent.".


    We are in roughly week 7 since I first reported this issue. This week my support contact asked me to compile more screenshots and descriptions of different ways that the Reports (Gift Summary and Detail, Solicitor Performance, Action Summary, etc.) are not looking right when delivered as PDFs via emails from scheduled Queues. In case anyone has any similar experiences (or clues!), I've shared some of these descriptions below... all from reports delivered this week or last!  My support contact thanked me for documenting these. He forwarded them to the "internal teams", and thought they'd help with diagnosis. I offered that an internal team member could reach out to me directly, e.g. for another screen-sharing session. Haven't heard back. We'll see what happens. It's a strange issue in that the bad output takes on so many different "looks", but I suspect it likely all ties back to a singular problem.


    1. Missing sub-total and total rows, report criteria

    2. In certain text columns, only the letters a, e, P, and g appear. In certain totals columns, only the numbers 0, 1, and 2 appear. 

    3. Missing title, subtitle, date, column headers and subtotal rows

    4. Page one shows only the date and page two has weird blotching and random character omissions in the report criteria.

    5. Missing title, criteria and column headings

    6. Triangular blotching and certain letters missing in title, sub-title and column headers

    7. Column headings intact but other parts totally missing or showing the letter “P” where ever it occurred in the report (and only when in upper case)

    8. One big square blotch, randomly-omitted letters, and lots of missing sections

    9. Tear-drop blotches all over the page

    10. Totally blank report

    11. Just the underlining below the (missing) column headings appear, that's it.

    12. Small narrow out-of-focus spots between characters in random places in the report results
  • I converted in September 2019. I have had the same issues with the email delivery of Queue and it took me forever (and a lot of work, screenshots, documentation) on my end to finally get an answer that it isn't working on this server and they are "working on it".  I agree that we are all paying for a product/service that just isn't what was promised.  My Finance Office is having the same issue in FE.
  • Thanks, Ann. Our FE people report some problems, too, but they seem minor compared to the messed-up pdfs we're seeing come out of RE Queue.


    BTW, readers might be interested to know that on Jan 16 a Blackbaud support emailed me to share that he...

    "...wanted to provide some additional information about the upcoming release of patch 18 for The Raiser's Edge. While I don't have definite dates to offer just yet, this is tentatively scheduled for the end of February. " 
  • To add fuel to the fire here, we have been running into the corrupted e-mail and locked queue problem for a couple of months here. It seems to coincide with when we updated to patch 17. We're self-hosted, though, which suggests this doesn't have to do with NXT or being cloud-hosted...


    I've started migrating as much as I can away from Queue and onto our ODBC connection as a solution until we get a better answer than "evaluating."
  • Thanks for that, Steven Cianciarulo. Interesting to read it seems to tie back to Patch 17 in non-hosted, esp. since they're talking about a new patch coming this month perhaps. Not sure an ODBC connection is an option for us.
  • Hallelujah, it seems the issue has been resolved! Coming to work this week,  I found that three PDFs delivered via Queue on Sunday showed NONE of the odd display issues that have been showing up in our Queue-delivered reports since last fall. Today, the Blackbaud service rep confirmed that patch 18 was applied to our RE databases (on Blackbaud servers) over the weekend. From then on, a total of seven scheduled reports have run via Queue and so far all have been spot in. And, so far we’ve noticed no adverse “ripple” effects from the new patch. So, very good news indeed... it appears we're back in business as far as Queue goes. Hoping it stays that way for a long time and that others find resolution to Queue issues via Patch 18.
  • Hey, Joel Raasch‍ thanks for the heads up on this!  We had actually turned OFF all our Queues and we were just ignoring the module (even though we paid EXTRA for it).... but based on your post we went ahead and tested a few of them out... We haven't run through all the different options, but yes, we're suddenly getting readable PDFs emailed to us on a regular schedule...


    Wow!  


    Still... Blackbaud generated A LOT of bad feelings at our organization over this. It's not a good look from a support perspective to constantly deny there is a problem, essentially stop communicating to clients that they are aware of the problem, and then continue SELLING the product even though it clearly isn't working properly.


    It should not have taken them 18+ months to work this out.... ?

     
  • You're very welcome, Tom. I'm glad that you saw my post right away and that the Queue issues at your site seem have been resolved by the patch. 


    P.S. Unrelated, but would you be so kind as to explain why and how you are able to get a name (like mine above) to appear in bold blue as a hyperlink in your posts. I assume there's a benefit for doing that and from what I've tried it doesn't seem to just happen when I type the other poster's full name in my post.
  • BTW, it was interesting. On Monday, I logged in to RE and had no idea patch 18 had been installed. There was no change to the log-in process, no welcome page talking about the new features/fixes, no new colors and no user settings suddenly set back to factory defaults, e.g. which columns appear where on each tab on a constituent record. My only clue that we might have been "patched" was that the Queues were looking right (at last). On Tuesday (Blackbaud was closed on Monday for President's Day.) Blackbaud confirmed we had the new patch. Then, yesterday (Wednesday), 5 days after the patch was installed, I was surprised anyone at our site logged in after about 9:00 a.m. saw all the things mentioned above in bold font. I assume this was somehow a delayed effect of the new patch, but I had no clue it was coming. Total surprise for me as database administrator (in spite of having asked to be notified before a patch was installed by Blackbuad and how it might affect us).


    So, all morning and into the afternoon yesterday  I and our Helpdesk staff found ourselves scrambling to answer calls from our 60+ RE users who were having trouble with lock-outs (due to "Administrator" being auto-loaded instead of the correct username), questions on where stuff went i.e. missing, narrowed, rearranged and re-sorted columns, why action and constituent default sets were not loading their default values, etc., etc.).


    I really wish there was a way to get notified BEFORE a patch just hits us hosted RE users. I understand that there was a way in the recent past. A few weeks ago a Blackbaud support rep even pointed me to a Knowledgebase article about how to subscribe to patch notifications. But I discovered on my own (and informed that rep) that as of last fall Blackbaud has discontinued subscriptions to patch notifications.Needless to say, I wish they hadn't discontinued those notifications. I'm hoping this post might help Blackbaud more fully grasp what a bad taste toward Blackbaud gets left in the mouths or our 60+ RE users when they get my apologetic group email explaining that Blackbaud surprised us again with an totally unannounced patch that is now adversely impacting all users in multiple ways.
  • Rachel Cavalier
    Rachel Cavalier Community All-Star
    Seventh Anniversary 500 Likes 500 Comments Name Dropper

    Joel Raasch:


    P.S. Unrelated, but would you be so kind as to explain why and how you are able to get a name (like mine above) to appear in bold blue as a hyperlink in your posts. I assume there's a benefit for doing that and from what I've tried it doesn't seem to just happen when I type the other poster's full name in my post.

    Joel Raasch and the name of the person you want, you should get a drop-down list of forum members to pick from. You pick the one you want and then their name shows as a hyperlink.

    Depending on their settings, they may get an email that someone has mentioned them on the forum and I think, at the very least, they get a notification in their forum messages. 

  • Thanks, Tom Klimchak‍ ! I would probably not have thought of adding "@" in front of the name. And, you're right... I did get an email in my inbox after each of your two posts today. Cool! 
  • Joel,

    Is it standard that after patches are applied the look goes to "factory settings"?  That is, "Administrator" is preloaded in user name and column order and selection etc go back to original?  Or is it something that happens with some patches only?  We are about to move to NXT & hosted andI know many of our users will still have to use database view for at least some of their work, so I want to be prepared.  I do recall that even as self-hosted we've had once or twice when "Administrator" got preloaded as username but other settings users made were not affected.


    Thanks,

    Tatyana
  • Joel Raasch
    Blackbaud surprised us again with an totally unannounced patch that is now adversely impacting all users in multiple ways.


    Very Annoying!
  • Tatyana Leifman:

    Joel,

    Is it standard that after patches are applied the look goes to "factory settings"?  That is, "Administrator" is preloaded in user name and column order and selection etc go back to original?  Or is it something that happens with some patches only?  We are about to move to NXT & hosted andI know many of our users will still have to use database view for at least some of their work, so I want to be prepared.  I do recall that even as self-hosted we've had once or twice when "Administrator" got preloaded as username but other settings users made were not affected.


    Thanks,

    Tatyana

    Good question, Tatyana Leifman‍ . I don't know the answer (yet). I do remember this same thing happening with at least some patches when we were self-hosted. Our Customer Success Manager is checking on whether or not these things only occur with some or all patches via Blackbaud hosting.


    He's also checking on whether or not we can be notified before a patch hits o, e.g. so I can warn/guide our users in advance. So far, he's learned/shared that the first step is to supply Blackbaud with our Tenant ID. We'll see what I hear back.

  • Is anyone else experiencing issues with their Queue not running since early March 2020?  Mine were working for a month or two and then stopped running (after, I am assuming an update to the server.)

    Thanks!
  • Keith Wilson 2
    Keith Wilson 2 Community All-Star
    Sixth Anniversary 250 Likes 100 Comments Photogenic
    Yes we are!
  • Keith Wilson 2
    Keith Wilson 2 Community All-Star
    Sixth Anniversary 250 Likes 100 Comments Photogenic

    Tom Klimchak:

    ...  We actually reached the "refund" stage because Queue was such a disaster.  


     

    Tom Klimchak‍ -- Did you ever get your refund? We were promised a refund last year for our Queue model, and we are still waiting. Calls to our Account Rep go unanswered or we're told to contact the Finance Rep who tells us to contact our Account Rep...

  • From what I remember we got to the Blackbaud Accounting department stage because our legal counsel was starting to look at what Blackbaud promised and what Blackbaud actually delivered.  But then, miraculously, Queue started working really well so we backed off and let the whole thing drop.  At some point after that we realized that while QUEUE was working well, we didn't have FTP access to all the crucial folders that Queue was using in the NXT Azure environment.  That started another whole round of support calls, chats and complaints.  


    Then I believe it stopped working again and by that point we'd had some changes in leadership and we were too preoccupied to fight anymore. 


    Strangely, we noticed Queue started working GREAT for us again right around late February of 2020.  But that seems to be right around the same time that it stopped working well for other people.  As of right now it works and runs reports for us on schedule... but we've spent so much time NOT using Queue that we generally don't rely on it much.  We have a few small processes that run (nightly exports, mostly) and now direct our users to use RE NXT reports or lists for most of their needs.  

     
  • My issue with Queue has always been it's "irregularity", and the fact it basically would only send to the hosted folder, from which I would have to send the file out to the user(s).  If I'm going to do that I might as well run the process myself and email it.


    I think it's an ancient, clunky, piece of software which inadequately provides a much needed service Blackbaud clearly believes is not needed in their NXT web view.  Or, as I have mentioned, it will be an even larger cost as another add-on Blackbaud has bought (reeher) or will buy, or advertise as the solution like ImportOmatic.


    Having said all of that, I do watch this thread for success stories... which continue beyond a few months. :) 
  • Hello Tom - Greetings from Fort Worth, TX!


    I am wondering if this issue you had with the Queue Module was resolved on your end? If so, what was causing the scheduler not to work?


    If not, has Blackbaud reach out to you and offer you a solution or alternative for this module to work properly?


    I appreciate all your feedback you can provide me at this time.


    Regards,

    Aura R.
  • Keith Wilson 2
    Keith Wilson 2 Community All-Star
    Sixth Anniversary 250 Likes 100 Comments Photogenic
    Aura Recuero‍, greetings from next door--Dallas, TX!? We were having issues with queue that took Blackbaud techs months to resolve. I'm not sure what they finally did, but like Tom Klimchak‍ "they miraculously started working".


    However, over the weekend, they stopped running again. ? I'm not sure what the problem is, but I will be reopening the Case with Blackbaud.


    I'm guessing your organization is having an issue also? 
  • Oy, sorry to hear Queue isn't working properly again... Our queue still appears to be working.  We don't do much emailing with it so we save out a file each night and use a secondary process to email the file to a few people, just as a daily test...


    We're actually starting to look around for some sort of automatic email program and would love any recommendations if anyone has one.  We've found that Queue can send run and send out PDF reports, can only run and save Excel files, and can't do much at all with Crystal Reports/Custom reports.


    We're Blackbaud hosted, so you'd think everyone would work the same but that doesn't seem to be the case.  


     

Categories