Trouble pulling large exports

Options
I am really struggling pulling large exports (over 85,000 records) from Raiser's Edge.  We just recently went hosted and I have not been able to successfully pull a large data export that our mail vendors need.  I have done all of the Blackbaud suggestions such as requesting an extension for time out, making the queries static,reducing the fields that I could, etc.  Nothing is working!  If anyone else has any other suggestions to help I would appreciate it.

Comments

  • What exactly is the issue? Are you timing out before it exports? Or getting a specific error message? We are also hosted and I regularly export files of that size (just to give you hope that it is possible!) 
  • Hi Kim-


    Start by exporting only the Constituent Ids of everyone in your entire database. If that goes prettty quickly, the most likely culprit is the complexity of your export. If you have a lot of calculated fields such as gift summaries it's going to take some time to complete those calculations.
  • I had this problem several years ago as a self-hosted client, and found the only way to address this is to get out of the GUI.  I've been told by Blackbaud that there is a memory cap on the standard interface we all use to tell RE what to do.  I have no idea if this will help in a hosted environment.


    We are not hosted, so we work around this by using scheduled Queue.  As a hosted client, I'm not sure how you would do that. You could also try using RE Mail to see if it will output your data for you (but we found that too limiting).  RE Mail allows RE to do it's processing behind the scenes out of the GUI.  We also have to output the files a .csv or Excel 1997-2003 (but row limits create a problem with the older version of excel).  While in theory, more current versions of Excel should work smoothly, I've run into bugs I can't explain or fix and Blackbaud hasn't helped solve our Excel issues yet.  


    Our files have grown even larger and eventually started causing trouble for Query.  So, to work around all of this I simplified our gift coding structure and changed campaigns, appeals and fund to streamline the output.  It was a lot of work, but today, it was worth the effort.


    Good luck and I hope you can figure out something that works for you.
  • You will need to use Queue and have the exports saved to Blackbaud's servers. Then download the files using an FTP client, such as FileZilla.  Also, if the export is large enough, I would recommend scheduling the queue to run at night or over the weekend.  I've tried to run a queue during work hours that would take nearly all day to finish, whereas running it after work hours will only take 30 minutes.


    I have several large exports I run, most weekly, some nightly.  I'm pretty much exporting our entire database twice - one for Reeher and another for a local data warehouse we use for reporting. There are times when the Queue will not finish processing for some reason. That might be due to system maintenance on BB's end.  Mailing from Queue doesn't work in hosted environment. 
  • @Claudia Sauers My colleague recently exported our annual appeal mailing list and it took 20 hours. Have you utilized the Queue module to assist in large exports? Thank you.

  • Austen Brown
    Austen Brown ✭✭✭✭✭
    Ninth Anniversary Facilitator 4 Name Dropper Photogenic

    @Kim Kruger - In my experience an export of this size while hosted can take upwards of 8+ hours, depending on how much summary information (calculations) needs to happen for each constituent. If you have multiple computers/accounts available you can break the export up into smaller pieces, and run the smaller sections simulanteously. As others have said, you could possibily utilize Queue to assist with larger exports as well.

  • Karen Diener 2
    Karen Diener 2 ✭✭✭✭✭
    Ancient Membership Facilitator 3 Name Dropper Photogenic

    @Kim Kruger:
    I am really struggling pulling large exports (over 85,000 records) from Raiser's Edge. We just recently went hosted and I have not been able to successfully pull a large data export that our mail vendors need. I have done all of the Blackbaud suggestions such as requesting an extension for time out, making the queries static,reducing the fields that I could, etc. Nothing is working! If anyone else has any other suggestions to help I would appreciate it.

    If Queue isn't an option for you, which does have an additional cost, I definitely recommend exporting your data in smaller groups. I have done two things in the past:

    • Break your audience up into multiple queries if you can. For instance, use your original query but update to only pull Organizations. Then save other versions that pull Individuals but only if their last name begins with A-M or N-Z. You may need more than two Individual queries, or don't even have Organizations in your query, but I think you get the idea. Once you're done, you'll need to manually copy and paste everything into a single file. A big pain, but I've had to use this method before.
    • Another option is to export only contact information first, such as name, addressee/salutation, and address info. Then export Constituent ID along with other information needed on the output file - maybe some gift information. Keep exporting logical sets of data, including the ID for all of them. Familiarize yourself with doing joins in Access, or using VLOOKUP in Excel, because that is how you'll need to piece the files together in the end.

    Once you get through this immediate project, I highly recommend talking with your direct mail vendor to understand why they need so much detail.

    If Queue is an option, consider it. It was a few years ago that I had it available, but it never worked even in a Blackbaud-hosted environment. It was tested A LOT by me, and Blackbaud support did a lot of troubleshooting too, so it isn't like I tried it once and gave up. I think we had it at a very large organization prior to that, but we could never get it to work their either, so I am personally not convinced that it is worth it. I have NO idea if Blackbaud would let you try it to make sure it is successful or not, but worth asking.

    Best of luck getting this done!!

    Karen

Categories