Missing Output Query in Duplicate Management Tool Forces Admin to Use Outside Lists!

Gerald Ollison · February 2016

*Beginning with an aside- I have also posted additional comments concerning this issue, as it relates to Imports- specifically New gifts- To the records of already-existing constituents. I don't wanna bore you with details but- This is usually done by importing Constituents first; performing a dupe check with said 'Output' Query; deleting the imported dupes; then finally using said query to export the already-existing IDs- so you can then re-import the new gifts on top of our already-existing accounts (outlined in magnificent detail in Knowledge Base Article Number: 41186)...

This used to be the very 'basic' routine... Anyway... Without the Output Query-which we used to have- you cannot effectively do this any longer. I have posted detailed comments about this issue to the 'Admin' category of our community.

There is an old saying about software upgrades- when administrative veterans refer to a vendor's removal of an old and valuable tool in patch or new software version, we jokingly refer to what is effectively a downgrade, as a "feature".

I have a very important issue to add around the De-dupe process/attribute process- Why the Output Query which used to existis still much needed... It's an institutional data hygiene kinda' thing...

In short- this whole issue recently came up because someone (as people in various departments are apt to do from time-to-time)- was maintaining their own separate contacts in an excel spreadsheet- Nearly 4K of contacts. (But if you frequently purchase lists, or perform list trades with other organizations you will also run into this issue).

We have a big money 125 Million dollar bond coming up for a vote for our instution in our area- And our department heads want to use those contacts for some of the campaign.

On initial visual of the data I knew immediately that most of those people were already in the database (and as it turns out only 20% of those were actually 'maybe' new people...)... Which means RE NXT's features have already been used to clean up their addresses, their constituent codes were labeled, their alumni data was already associated, and they had many additional Attributes already associated with their constituent accounts, etc... In our existing database...

In short, all of the wonderful advantages our fine NXT Database system utilizes to segment our constituency in hundreds of ways, were already being utilized to track most all the 4000 people which were also listed in this secretly maintained 3rd party spreadsheet...

So, why use the stupid spreadsheet..?

No... Of course, you wanna use the already-existing 'clean' data, right..? All I'd have to do is import all the spreadsheet, perform a De-dupe with an 'Output Query' which shows the imported spreadsheet, plus my already existing cleaned-up constituents!

All I have to do now is use the Dupe Output Query to help Globally Add an attribute related to the new campaign to every 'clean' dupe constituent already existing in my database... and then delete the imported imperfect ones...

And there- I'd have all those same people now marked w/o importing 80% of them- and they've already been segmented and are good to go...

But guess what..? I can't do that now... Instead of being able to use RE NXT, with all the advantages it has, to help me with my data, I now I have been forced to use the old stupid spreadsheet...

I think this is the opposite of what Blackbaud's intentions are... Instead of using my already clean data (which I've maintained with RE's help), I am forced to use to old, unclean data... or worse... an entirely different database on campus (and pointing me to another databse would be the exact opposite of what Blackbaud would want, wouldn't you think?)

These secret contact sheets come up all the time... From different departments, in a large institution... Colleges are like small cities...

Anywho, there's another very important example of how the now missing output query has massively hurt my productivity- and if you have another solution, please let me know...

Please run this up the flagpole.

Thanks. Peace.
Jerry'O

Cathleen Mai · February 2016

I feel your pain. Try Importacular (from zeidman) the import of Constituents is free and has a duplicate search that you can customize.

I have heard good things about Importomatic (Appomatic software), but you would need to pay for it. Hope this helps.

Gerald Ollison · February 2016

Thank you, for your suggestion... I would really like for Blackbaud to pay attention and reassess their removal of a feature virtually everyone other database has, and which effects both routine maintenance, and basic gift import functions... As I stated before (and this happened to or organization too), many who are using older RE versions still don't know about this change, and we're told we'll still have the same abilities available on the back-end, when we upgrade to a new version, but better... It is no small surprise then- to discover you can no longer import new gifts to existing accounts (basic), nore cleanly compare outside traded/purchased/departmental lists against your existing members (routine)... Not an exageration- This is a crippling disturbance to our organization's work-flow...

Nicole S · February 2016

This is somewhat unrelated, but this is another problem that I have with this tool and I would like BB to review it: after merging records, they just GO AWAY. The newly merged record doesn’t open for the user to review/cleanup and we all know that automated processes are not foolproof. In fact, because of this issue, I can’t really let others use the tool because I know merged records need review and it is so easy to skip this step when using the tool.

duplicate constituent merge utility - open record after merge
When using the duplicate constituent merge utility, I'd like to have record open after merging.
Anytime you do something by an automated process a certain amount of manual human cleanup is required. When merging records through the constituent record, the merged record opens and you can review it. The merge tool just merges the records and removes them from the list. You have to write down the records or remember them somehow so that you can go back in for clean-up.

Gerald Ollison · February 2016

Hear, Hear!

Anne Herbert · February 2016

The lack of the output query is a real concern - we are supposed to go hosted and to NXT in the next month. What else is missing?

Gerald Ollison · February 2016

Oh... Wait... Ignore my first response... I misunderstood... I somehow thought you were BB support, and that you were saying the problem was being fixed... Oops!

I am still cleaning up some conversion things, so and haven't seen much else in the way of surprises... So far, I mostly like the product...

Which is why I am soooo disappointed at this major oversite... And I do consider this Major... (can't import new gifts to existing accounts without an outside app- C'mon!!!)

So please be a squeaky wheel with your BB Reps,Tech and other contacts about this...

Kind regards.

Jerry'O

Emmet O'Conlon 2 · February 2016

I agree it's not a good trend that Blackbaud has been engaged in lately: removing features/tool/functionality.

On this particular issue, what we do is:

1. Generate the report and export it to Excel

2. Delete everything but the Constituent IDs

3. Do a "blank" import of just the Constituent IDs, with the option to create a query checked.

This creates a query for us of all the potential duplicates. Not as good or as useful as the old functionality--using it can be a bit clunky, frankly--but it's better than naught.

Hope this helps!

Gerald Ollison · February 2016

Emmet-

Thanks for response... I am going to re-write and paraphrase your work-around to make certain I understand what you are saying (and add to it). I have a list of 4k potential dupes... I will then:

A) Import all except the dupes (they go to the exception file)- What is imported this round are probably my 'New people'...

I then Import all the Exceptions (w/o dupe check and exceptons), with Import Output Query of what are now Known dupes;

C) I run a dupe report, and export the results as a spreadsheet;

D) I use the spreadsheet to sort out which Members were newly added to RE (by add date/ID #'s), and which Existing are already in the database;

E) I use the Ouput Query (in 'B') above to delete the New Member Dupes in RE.

F) I also delete all but the pre-Existing Member Constit IDs from my Spreadsheet.

G) I can then use the pre-Existing spreadsheet IDs to update those Existing Members with either new Attributes (...or gifts even)..?

. . ?

Is this what you mean..? It sounds fairly reasonable... Like it might work... I will have to try this...

You say it's a "bit clunky"... What problems should I look for..?

Are you (or anyone else) using this method to import new gifts to pre-existing accounts?

Please let us all know.

This is a great help if this works!

Jerry'O

Emmet O'Conlon 2 · February 2016

We're not using this to import gifts. We have A LOT of potential duplicate constituents already in our database, so we've been using this method to try to eliminate the "more important" duplicates first.

What you propose sounds like it should work, but I never know myself until I try it. Maybe a small test run with a small record count would be in order. Good luck!

Gerald Ollison · February 2016

I underdstand- same problem here- and this method doesn't look particularly stable for import purposes...

I appreciate the inspiration though...

I wish Blackbaud would be more responsive on this issue...

Thanks, again.

Jerry'O

Jarod Bonino · February 2016

Gerald,

First, let me apologize for the lack of response to date from us here at Blackbaud. We certainly aim to be active and responsive in this community and our lack of ackknowledgement to this particular thread for this long is not good enough. We can, and will, do better going forward.

Now on to the content of the post itself. This is certainly a complex issue that you are describing and I want to answer what I can for now and tell you what we are doing in the coming days to address the rest.

I understand that you are concerned with the removal of the option to generate an output query of duplicate records when using the Duplicate Constituent Management Tool feature. First, some clarity on when and why this was done. I saw a few comments that suggested the change may have been RE NXT related, so I wanted to clear that up. The change was made in RE 7.93 and actually does not have anything to do with any changes made to support RE NXT development.

Now on to why the option was removed. During the implementation of the Duplicate Constituent Management Tool, the (perhaps incorrect) assumption was that the move from a read only report to an interactive utility would eliminate the need/want of the output query. The assumption during development was that the query was primarily (or solely) being used to navigate to the records individually to handle de-duplication. Since de-duplication is handled within the utility itself, the usefulness of the query was seemingly eliminated. I am happy to review this decision with the development team to see what, if anything, can or should be done about it.

In regards to the specific immediate issues this change has caused for you:

Problem 1: Unable to import gifts effectively from an import file that includes gifts from both existing and new constituent records

I have reached out to our support team to make sure we update KB 41186 to include a way of accomplishing the task without the use of the query in question.

Problem 2: Unable to import large lists of constituent records and effectively de-dupe afterward to ensure the “clean” record remains

I apologize if I am missing an obvious point on this, but I’ll play the “new guy” card and ask the obvious question, if you were to import the records and just use the Duplicate Constituent Management Tool afterward, would this not accomplish your goal? If not, is it because you have other pre-existing duplicates that you do not want to consider as a part of that tasks? Or is it difficult to differentiate between the “clean” and “imperfect” duplicates in the tool given the information it surfaces?

I understand that your concerns extend beyond these two specific issues, but while we discuss options I want to make sure that any immediate concerns are addressed if possible.

I will reply again to this post with an update early next week.

Thanks,

Jarod Bonino

Gerald Ollison · February 2016

First, thank you for the response...this is very much appreciated... I understand you are out there working to perhaps make things better- not just efficient, but also hopefully more effective... This is all in the spirit of making the product better, I'm certain. I am glad BB is occasionally responsive- and asks input from those of us who are actually in the trenches, attempting to make some otherwise impossible procedures work...

That said... Here are my responses, and some general reflection:

<<Problem 1: Unable to import gifts effectively from an import file that includes gifts from both existing and new constituent records
I have reached out to our support team to make sure we update KB 41186 to include a way of accomplishing the task without the use of the query in question.>>

Hey, just put the query back in... End of problem.. I and others would be happy to wait for the next big thing- but until then please let us maintain the ability to continue doing basic import stuff...

As for the article itself- If you have another method, please tell us know right here, I don't wanna wait... I attempted to create an outline of one possible work-around here in this thread, with another user's help [and if you steal that you better credit us!

]... but seriously- on reflection, there are way too many flaws with this method... Otherwise, in addition to the article- it still appears like you have to update the database itself...

<<Problem 2: Unable to import large lists of constituent records and effectively de-dupe afterward to ensure the “clean” record remains
I apologize if I am missing an obvious point on this, but I’ll play the “new guy” card and ask the obvious question, if you were to import the records and just use the Duplicate Constituent Management Tool afterward, would this not accomplish your goal? If not, is it because you have other pre-existing duplicates that you do not want to consider as a part of that tasks? Or is it difficult to differentiate between the “clean” and “imperfect” duplicates in the tool given the information it surfaces?>>

Firstly- My data is as clean as newborn baby's...

Don't you every imply my data is sub-par... You don't get away with this that easily...

The problem with this is two-fold:

1) The dupe management tool is incremental/ad hoc... How many hours do you want us to spend merging 4K of duplicates- one at a time..?

2) Also, where is the method to identify the merged duplicates afterward..? So I can export the existing Info for mailings or whatever? I suppose I could query 'Date Last Updated', if I and everyone else I worked with aren't in any of the accounts that day... Even with the old output query scanning/sorting 4K of dupes could take a couple of days... I can't exactly stop everyone else from working while I do this... this simply isn't effective...

Before the missing Query- it was so simple... all we had to do was use the output query to globally mark all the existing dupes with an attribute... and I could immediately turn around and export the 'clean' existing Constituents for a mailing, or work for days doing other stuff if need be, without interfering with co-workers using the database, whatever... We cannot do this now...

This really is bad...

Also- I know this change is not directly related to NXT; however, I was (and many like me still are) working in v7.92... I think... And then I moved to this community college who was switching from Jenzabar, to RE for the first time... NXT represented some forward thinking in the application's front-end user friendly interface- But NXT is also represented with the promise that the back-end still possesses most all of the components which make up the functionality of prior versions...

Who (especially those of us familiar with prior versions) woulda' thought to assume the ability to import new gifts to existing accounts would be in anyway lost..? And in such a forward thinking product?... That's just a 'basic' database gimme...

I know there's OLX, and we can create our own online gift interfaces, etc... and you are working to make RE a broader package...

But we on the ground don't always get to use RE products exclusively- Plus we aren't able to make 'the call' for every department in a large institution. I, and many like me out here- are still interfacing with 3rd party Wedsites, Credit Vendors, Auction Services, Survey Tools, Scholarship Software, and much, much more (one department at our college is actually using SalesForce)...

Plus there's always that person, sometimes a Chief Executive- who wants to send an immediate mailing to their own list of stakeholders and VIPs (but they may already be in the database, as clean data...)...

Plus- We not only collect gifts... but have all kinds of additional data we need to import into existing accounts... to do this, we need that query to get unique IDs... The absence of this simple query effectively hobbles us- period.

Every six months, I will have approximately 3,000 Student Alums to import into my database. We are a community college, and our peculiarity is that, unlike universities, people are constantly coming back to get new professional certifications, or just take all kinds of adult education courses... How do I effectively update existing Alumns' accounts with New 'Primary' Alumni Information w/o existing IDs..?

You took this ability away...

This makes the process inefficient, very time consuming, and in general non-effective...

Thank you, again, for your time.

Jerry'O

Jarod Bonino · February 2016

Gerald,

Thanks again for your insight. The team is still looking into the need and potential solutions here and I plan to give you a more detailed update within the week, but our support team passed on a blog post that seemed particularly relevant that I wanted to make sure you saw:

How To Import New And Existing Constituents Without Import/Constituent IDs

https://community.blackbaud.com/blogs/10/754

Again, I am not suggesting that this is the magic bullet or that we are done looking at this, but I wanted to share it asap while we continue to look into the concerns. If you do get a chance to read it (and maybe try it), feel free to let me know if it does eliminate any of the concerns you have mentioned in this thread.

Thanks,

Jarod

Gerald Ollison · February 2016

Jarod-

- Import new gifts onto existing records.
- Globally flag (possible) known duplicates for data management.
Two of the simplest things any database should do.

I understand it will take time to address the issues which have been caused by Blackbaud's removal of the ability to create a Duplicate Output Query- Which allowed us to perform Gift Imports, and additional functions, almost exclusively from within Raiser's Edge all by itself... without much assistance from outside software.

And I feel I must stress that although my online tone in these messages may appear combative, please understand this is not my intention... However, my immense frustration regarding this issue is more than casual...

So, all that said... though I appreciate the provided workaround, I feel the need to re-state an obvious point- namely that the

work-around process laid out as provided actually expresses the inherent problem:
The bulk of the burden and abilities- to effectively import gifts and additional data to update existing constituents records, no longer resides in Raiser's Edge itself, and has been shifted to outside software (in this case namely, Excel).

You yourself mentioned about the provided workaround that, "I am not suggesting that this is the magic bullet or that we are done looking at this...". Correct, but I want to re-detail some of the complications.

The inefficiencies of this predicament are obvious and multi-fold- This means that:

- In order for your clients to continue to perform these basic import functions, we now have to rely on Outside Software alternatives to RE, and formulas provided by those outside vendors, and track complications caused by their updates, in addition to yours;

- With the Output query, we could easily click on and view/compare all duplicate constituent details within RE itself; those details are no longer a click away when utilizing outside software like Excel (...of all things! God, I can't believe Blackbaud steering us to any outside vendor, let alone Microsoft for our 'simple' Imports!)...

...First we'd verify all data within RE itself- and only then export the (already existing and clean) constituent data!

- Since the dupe Constituents would already be in RE itself, I could efficiently use RE's existing Global functions to add/change Attributes or other updates on the existing dupe constituents- all available within RE itself, power at our fingertips...

... Such global changes would now require an additional 'Import Step'.

- There is no constituent Import number limit I have ever run into when utilizing the Duplicate Output Query (this work-around has inherent limits)... The conventional process was about equally as efficient as when I imported 500 constituents, 5000, or 15,000...

I said previously that every 6 months I receive list of 3000-plus College students, many of whom whose Primary Alumni information I must update for existing Constituents... With the Re-Import of every 500 dupes (after using the macro formula in Excel), this is not going to go well...

With the gift import from our coming auction, there will be similar issues...

You asked early on, if my problem with using your alternative suggestions was if I already had too many existing duplicates- if that might create complications with your suggested methods..?

I responded my data was very clean... Well, it is... for now... I am extremely worried that this will no longer be the case, now that my data-checking capabilities are limited to the work-arounds provided...

I used to feel empowered by this process... no longer...

Again, not attempting to be combative, but just expressing what processes we deal with daily.

- Import new gifts onto existing records.
- Globally flag (possible) known duplicates for data management.
Two of the simplest things any database should do.

Thanks, again.
Jerry’O

AnnMarie Schell · April 2020

Have all these issues been resolved? I just got Kimberly Coughlin's post today! https://community.blackbaud.com/blogs/13/6768

Missing Output Query in Duplicate Management Tool Forces Admin to Use Outside Lists!

Comments

Categories