Database Merger - what I've learned

Joanne Felci 2

Sharing this to vent to the group that will understand...plus maybe I can help someone avoid some of this extremely PAINFUL learning


Since last September I have been working on preparation of merging two existing Raiser's Edge databases - one of which was self hosted and one hosted (we merged into the hosted database)


Initially we though there might be 100 or so duplicate constituents between the databases - there are over 5,000 (out of 35,000 records we were merging into the parent database)


The number of duplicates has made every other import more complicated/time consuming as everything needs to be matched back up, which leads me to my first learning:


If you are a hosted Raiser's Edge database, just use Blackbaud to do the merger, even if you think you will have a big mess to clean up afterward. I'm still going to have a mess but all these individual imports are KILLING me and taking well over a month to get through.


At the very least ask the vendor - can do all these imports by a script? Or is it all manually effort...if manual plan accordingly. I truly had no idea what I was getting into, but I think a big part of it is that I didn't realize the vendor we went with could not access the back end of RE since it was hosted.  And we were way too far down the track before we realized that so it is what it is


Actually it isn't the imports that are killing me it is the exceptions, which leads me to the second learning: even after testing your import keep your import files small so you do not have to hunt/peck through 3,000 or more rows looking for which rows actually went into the dang database


My personal pain right now is Action Notepads - first the Notepad IDs are not unique - WTH?! And who knew there was a limited number of digits you can use?!  I do know that now - I think the limit is 20 characters.


I really should not complain, I used to be bored - I'm not bored anymore...


If I EVER do this in the future - and I cannot imagine EVER doing that right now - I'd use Blackbaud or Omatic or whoever the heck can do a back end script for it all or at least the majority of it - SIGH!!

Carrie Aranda

Thank you for sharing your experience!  We are preparing for a possible merger and you just gave me a few more things to add to my list of implementation questions.

Catherine Burns 2

Hi Joanne. Thank you for your useful post.


It sounds like you might not have ImportOmatic. I highly recommend the Omatic products. I merged a database of 15,000 records from a different database into a hosted RE database a few years ago using ImportOmatic and MergeOmatic. There were so many duplicates (about 5000 of the 15,000) that I actually turned off ImportOmatics' duplicate search so I could clean them up later with MergeOmatic which we also had. I did this for time management but also because MergeOmatic gives fewer false duplicate matches than ImportOmatic as you can control the duplicate criteria better.


However if you are using ImportOmatic for your duplicate searches one of the great features is that you can "Skip rows requiring user interaction". This creates an exception file of all the rows with possible duplicates that you can come back to later and go through one by one. It is a great time saver and means that you do not need to limit the size of you import files too much because you know exactly which records were not imported. But importing the possible duplicates is still painfully slow as you need to consider each list of possible matches.