Marking records as not a duplicate to exclude from the duplicate management tool
OK so I have a number of records that have the same email address but are not duplicate records, Siblings, Parent-child, Partners, you get the picture.
I don't actually use the dupe managment tool to process the possible dupes as it is verry time consuming. I export the CN's from it then use thoes to create an excel spredsheet that I can use formulas to match on certain criteria such as email.
The problem is that when I get a NON dupe based on a matched email I do nothing, then next time I run the duplicate management tool all of the previously checked and confirmed NON dupe record are there again.
I want to be able to tell the duplicate management tool that certain records are NOT a match and to skip them in the future.
I have acces to import/export functions as well as importomatic if that helps.
Comments
-
Can you use the duplicate management in RE for just those records you know are not dupes? For instance, if it flags George Smith and his son Frank, can you just find George/Frank in the duplicate list and mark them as not duplicates? That seems to be the only way I've found to have them not pop back up on the next run.0
-
Amy - I think her question is about HOW one does that? Can you tell her how to do that?0
-
Editing my post as I see I totally answered a different question from the one that was asked.
0 -
Amy Dana:
Can you use the duplicate management in RE for just those records you know are not dupes? For instance, if it flags George Smith and his son Frank, can you just find George/Frank in the duplicate list and mark them as not duplicates? That seems to be the only way I've found to have them not pop back up on the next run.Hi Amy, Great idea but the problem is that the Duplicate Management Tool has not always matched the specific constituents together.
IE: Three constituents A, B & C The tool has matche A with B because of similar last name and then a diferent match has matched B and C because of similar First name. A and C have the same email but have not been matched together in the DMT so I cannot mark them as not a dupe.
Bradley.
0 -
I *think* there's a work around, but it's a bit slow. If you edit one of those records individually, it often comes up with a pop-up window when you try and save it, telling you it's a duplicate and giving you the option to continue with save/ move to the duplcate record / mark it as a duplicate. I believe if you do that it marks them as a duplicate in the same way that the main tool does and so it shouldn't appear when you run the main tool.
There's a challenge, obviously, of making the two records sufficeintly similar in the first place, without forgetting which is which (and it would be time consuming), but you might find with some of them that if you try and save them without even making any changes the dupe tool will pop up like this anyway.
My understanding is that somewhere in the back end of RE there is a table which has a list of pairs of records which are not duplicates, and both tools work with the same one.
Matt1 -
Bradley Henry:
Amy Dana:
Can you use the duplicate management in RE for just those records you know are not dupes? For instance, if it flags George Smith and his son Frank, can you just find George/Frank in the duplicate list and mark them as not duplicates? That seems to be the only way I've found to have them not pop back up on the next run.Hi Amy, Great idea but the problem is that the Duplicate Management Tool has not always matched the specific constituents together.
IE: Three constituents A, B & C The tool has matche A with B because of similar last name and then a diferent match has matched B and C because of similar First name. A and C have the same email but have not been matched together in the DMT so I cannot mark them as not a dupe.
Bradley.Ugh. I guess I assumed that the duplication tool would still find them. Witchcraft, maybe?
1 -
Ironically, I was just about to go in and try out this tool when I saw this thread. Used the count to finish up a proposal for data scrubbing that has now just been shot down as having no ROI. I would love to know if I was even close on my estimate of how long it would take to review and resolve the 4122 potential duplicates in our database, but it appears that we won't be doing any of that work now.
Probably won't help with regard to the DMT in RE, but I have a Relationship Type of "POSSIBLE DUPLICATE" and one of "DUPLICATE RECORD" so that I can link records together as I come across them and then someday come back (when I have time!) and resolve them. It's also come in helpful when I see that Relationship Link as I'm doing other work...and I have a Business Rule for this situation.2 -
Matthew Page:
I *think* there's a work around, but it's a bit slow. If you edit one of those records individually, it often comes up with a pop-up window when you try and save it, telling you it's a duplicate and giving you the option to continue with save/ move to the duplcate record / mark it as a duplicate. I believe if you do that it marks them as a duplicate in the same way that the main tool does and so it shouldn't appear when you run the main tool.
There's a challenge, obviously, of making the two records sufficeintly similar in the first place, without forgetting which is which (and it would be time consuming), but you might find with some of them that if you try and save them without even making any changes the dupe tool will pop up like this anyway.
My understanding is that somewhere in the back end of RE there is a table which has a list of pairs of records which are not duplicates, and both tools work with the same one.
MattHi Matt,
I guess the biggest issue is that what if you get distrated and forget to change the record back. One idea that I am considering is to create an atribute where I can put the CN of the NOT a dupe record. this way when I do my export I can include this attribute then delete those records straight away. I am thinking this is the way I may have to do it.
Bradley.
0 -
Jen Claudy:
Ironically, I was just about to go in and try out this tool when I saw this thread. Used the count to finish up a proposal for data scrubbing that has now just been shot down as having no ROI. I would love to know if I was even close on my estimate of how long it would take to review and resolve the 4122 potential duplicates in our database, but it appears that we won't be doing any of that work now.
Probably won't help with regard to the DMT in RE, but I have a Relationship Type of "POSSIBLE DUPLICATE" and one of "DUPLICATE RECORD" so that I can link records together as I come across them and then someday come back (when I have time!) and resolve them. It's also come in helpful when I see that Relationship Link as I'm doing other work...and I have a Business Rule for this situation.How do they determine ROI on data cleanup???
1 -
Jen Claudy:
Ironically, I was just about to go in and try out this tool when I saw this thread. Used the count to finish up a proposal for data scrubbing that has now just been shot down as having no ROI. I would love to know if I was even close on my estimate of how long it would take to review and resolve the 4122 potential duplicates in our database, but it appears that we won't be doing any of that work now.
Probably won't help with regard to the DMT in RE, but I have a Relationship Type of "POSSIBLE DUPLICATE" and one of "DUPLICATE RECORD" so that I can link records together as I come across them and then someday come back (when I have time!) and resolve them. It's also come in helpful when I see that Relationship Link as I'm doing other work...and I have a Business Rule for this situation.Maybe I am totally misunderstanding, but wouldn't it still be worthwhile to go through the 4122 potential duplicates in the Duplicate Management Tool anyway? Bradley's complaint, as I understand it, was not that the Duplicate Management Tool doesn't work, but that it is time-consuming so he isn't marking them non-dupes within the Tool itself, so they keep showing back up. He is looking for a way to mark them non-dupes en masse. I agree that the Tool takes time, but once you take the time to go through it the first time, the maintenance is minimal. Personally, I find the Tool a better option than searching in Excel, because Excel can't search on fuzzy data very well. On one particular import project, we hired a firm to do an Excel purge for dupes between our House list and an import file of new constituents. The Excel purge caught most of the dupes, but several hundred still remained when I began looking through them individually. Misspelled names, moved addresses, and changed emails caused them to be missed in Excel's search criteria. The advantage of the RE Dupe Tool is that it uses fuzzy data to compare potential dupes. Thus a donor named Claeys is flagged as a potential dupe to another fellow named Clarys.
Also yes, it's possible the Tool may only pick up 75% of the possible dupes in your system, but once you go through them and merge them/mark them "not duplicates", that's still several thousand less dupes in your system who are potentially receiving duplicate mailings and wasting your organization money in postage.
2 -
Amy Dana:
Jen Claudy:
Ironically, I was just about to go in and try out this tool when I saw this thread. Used the count to finish up a proposal for data scrubbing that has now just been shot down as having no ROI. I would love to know if I was even close on my estimate of how long it would take to review and resolve the 4122 potential duplicates in our database, but it appears that we won't be doing any of that work now.
Probably won't help with regard to the DMT in RE, but I have a Relationship Type of "POSSIBLE DUPLICATE" and one of "DUPLICATE RECORD" so that I can link records together as I come across them and then someday come back (when I have time!) and resolve them. It's also come in helpful when I see that Relationship Link as I'm doing other work...and I have a Business Rule for this situation.How do they determine ROI on data cleanup???I don't know...I can't follow the logic, and there are additional conversations happening without me, so I'm more or less just getting the end-result decision. My original proposal had a list of 10 high-level categories of cleanup, and only 3 made it past the first consideration. Sigh.
0 -
Jen Claudy:
Ironically, I was just about to go in and try out this tool when I saw this thread. Used the count to finish up a proposal for data scrubbing that has now just been shot down as having no ROI. I would love to know if I was even close on my estimate of how long it would take to review and resolve the 4122 potential duplicates in our database, but it appears that we won't be doing any of that work now.
Probably won't help with regard to the DMT in RE, but I have a Relationship Type of "POSSIBLE DUPLICATE" and one of "DUPLICATE RECORD" so that I can link records together as I come across them and then someday come back (when I have time!) and resolve them. It's also come in helpful when I see that Relationship Link as I'm doing other work...and I have a Business Rule for this situation.Jen,
If it is any comfort, the duplicate management tool counts both records as duplicates, so that when you fix one, the duplicate count drops by two, so you really only have 2000 duplicates.
5 -
We use the duplicate management tool in RE and try to keep the count to zero but I feel that it doesn't get everyone. So once a month I export all the individuals in the system and look for duplicates by FN, LN, Add1 or LN, Add1. Once I find the duplicates or household duplicates, I import a solicit code into these records so that I could find them in RE. Then I clean out the duplicates and if I find that they are not duplicates (family or spouses) I mark their one of the records with an attribute. Here is what we use:
- Is NOT Head of Household/Should NOT receive direct mail
- Not a duplicate household
Seems like a lot but this is how we were able to get the duplicate count down to zero and the cost of our mailing down.2 -
Jen Claudy:
I don't know...I can't follow the logic, and there are additional conversations happening without me, so I'm more or less just getting the end-result decision. My original proposal had a list of 10 high-level categories of cleanup, and only 3 made it past the first consideration. Sigh.I feel for you! At our org, I pretty much get to do whatever cleanup I want in the system, as long as I can schedule my own time for it. Would hate to have the level of red tape it sounds like you have to go through.
0 -
F Murray:
Jen Claudy:
I don't know...I can't follow the logic, and there are additional conversations happening without me, so I'm more or less just getting the end-result decision. My original proposal had a list of 10 high-level categories of cleanup, and only 3 made it past the first consideration. Sigh.I feel for you! At our org, I pretty much get to do whatever cleanup I want in the system, as long as I can schedule my own time for it. Would hate to have the level of red tape it sounds like you have to go through.Oh, I'm more than welcome to do any scrubbing I want...so long as it doesn't take time away from any of my other work, which means from home on my own time. (I'm salaried, so no overtime.) The from home is great, the on my own time is not. I am no longer willing to put in personal time (or at least not this much) on work projects unless there's a really good ROI for me, personally. Like time spent in the Community. A lot of that time is during the work day, when more people are actively posting, and then I spend personal time catching up on work projects (many of which are done more efficiently from home where there aren't constant interruptions).
1 -
I find the problem is working out whether they are dupliates or not. A lot of our constituents only have initials not first names and so often we have someone on the system with first name and someone else with the same initial. And then there's info on both constituents so they need merging. Getting through 4000 of these will be a major hassle.
Matt1 -
I agree -- sometimes even figuring out if you've got a dupe or not is hard enough. And then trying to figure out which is the cleaner/more correct record when you've got different addresses or spellings.
DO NOT under any circumstance spend your personal time scrubbing data or anything else for work. You will only grow to resent the job. If it needs to get done, do it on the clock or not at all. I know that sounds harsh, but thems the breaks, as they say. Sounds like a catch-22. You need clean data to do your job, but you can't clean the data because you're doing your job... I have found that there is typically a slow time of the year when I can work on scrubbing. And even then it doesn't all get done. It's a never ending project, which is another reason why you should't do it in your personal time. It's not like you're going to spend a few hours and BOOM you're good to go. There is always more to do. Doing work on personal time gives the illusion that it's possible to get it all done. Your personal life will suffer. Before you know it, you will be a sad, downtrodden, put upon DBA that the rest of the office will dump more and more work on.5 -
For Matthew: we have gotten our hands on LexisNexis to cross reference names, addresses and sometimes ages with our costituents having either common names (i.e. John Pederson) or just initials (M. Johnson). The address usually allows us to identify the individual and fill in the lack of details and LexisNexis has home addresses from years prior. This also gets more hits using the duplicate management tool! We also find deceased constituents this way...1
-
Tracy Morgan:
For Matthew: we have gotten our hands on LexisNexis to cross reference names, addresses and sometimes ages with our costituents having either common names (i.e. John Pederson) or just initials (M. Johnson). The address usually allows us to identify the individual and fill in the lack of details and LexisNexis has home addresses from years prior. This also gets more hits using the duplicate management tool! We also find deceased constituents this way...We currently have a Lexis-Nexis subscription, but are going to need to drop it because of cost. Blackbaud has told my boss that their tools (like Email Finder, Phone Finder, Deceased Record Finder, Age Finder) can completely replace any need for L-N, but I'm not thinking that's exactly true. I do a lot of one-off searches looking at potential relatives, etc. But it's not worth $7k+ a year, so I'll have to figure out where else to get at least some of that data...
0 -
We use Lexis/Nexis (Accurint) but don't pay for a yearly subscription. We pay per search -- it's about 50 cents for an advanced person search, 40 cents for an email search, $1 for a relative search. Don't have the price for a deceased person search off the top of my head, but this has been an incredibly inexpensive way for us to get the info we need.1
-
Hi Folks, if you're struggling with dupes in RE -please- try the 30-day free trial of MergeOmatic (or at least watch the six minute video here). To answer OP's question, you can easily mark records as not dupes (as well as see them all/change some back again if you later decide they are dupes). It also does a much better job of finding/ranking/displaying possible duplicates than anything else. If you like, one employee (or volunteer) can review the dupes and queue them for merging, and then a supervisor can review the queue before merging in batch. It also creates a confidence score for the matches, so you can sort and filter to work with the "low-hanging fruit" first. Finally, it archives practically all the data from the merged record so you could go recreate it if you ever had to (and you'll have a full record of who was merged into who when). We spent years of developer time working on this solution, if you're glad to see someone working on this pervasive problem, please click "good answer" above. I look forward to hearing your feedback!
Jeff
3 -
Hmmm the free trial does sound tempting. Thanks.
Matt0 -
I just received a copy of the Blackbaud Update email and it mentions KB article 52635 which has helped me work out another option for duplicates.
Basically in Business Rules, under "Duplicates" you can tick:
"When displaying constituent or duplicate search results
[] Display only constituents that the user has rights to view"
Which means that if you use Constituent Code to limit the range of constituents that a user can access, and then get them to run the "Duplicate Constituent Management Tool", it will only search for duplicates amongst those constituents.
So if you want to only search among a group of constituents, you just have to temporarily assign them a temporary constituent code, limit a user's access to only those with taht code (and you could also make a special user account for this purpose) and then get them to run the Dupe Cons. Management Tool.
I think this might be quite useful for us. Hope it helps some of you as well. It's another tool for the kit anyway...
Matt1 -
I have created a Constituent attribute called Duplicate Check. It has a table associated with it, but only one currently used table entry: Not a Dup - Skip. I create an output query from the duplicate report and add the criteria that this attribute is missing from the records. It successfully eliminates those records that I have already confirmed as non-duplicates. I then proceed to export additional information for the remaining constituents and examine in Excel.0
Categories
- All Categories
- Shannon parent
- shannon 2
- shannon 1
- 21 Advocacy DC Users Group
- 14 BBCRM PAG Discussions
- 89 High Education Program Advisory Group (HE PAG)
- 28 Luminate CRM DC Users Group
- 8 DC Luminate CRM Users Group
- Luminate PAG
- 5.9K Blackbaud Altru®
- 58 Blackbaud Award Management™ and Blackbaud Stewardship Management™
- 409 bbcon®
- 2.1K Blackbaud CRM™ and Blackbaud Internet Solutions™
- donorCentrics®
- 1.1K Blackbaud eTapestry®
- 2.8K Blackbaud Financial Edge NXT®
- 1.1K Blackbaud Grantmaking™
- 527 Education Management Solutions for Higher Education
- 1 JustGiving® from Blackbaud®
- 4.6K Education Management Solutions for K-12 Schools
- Blackbaud Luminate Online & Blackbaud TeamRaiser
- 16.4K Blackbaud Raiser's Edge NXT®
- 4.1K SKY Developer
- 547 ResearchPoint™
- 151 Blackbaud Tuition Management™
- 1 YourCause® from Blackbaud®
- 61 everydayhero
- 3 Campaign Ideas
- 58 General Discussion
- 115 Blackbaud ID
- 87 K-12 Blackbaud ID
- 6 Admin Console
- 949 Organizational Best Practices
- 353 The Tap (Just for Fun)
- 235 Blackbaud Community Feedback Forum
- 55 Admissions Event Management EAP
- 18 MobilePay Terminal + BBID Canada EAP
- 36 EAP for New Email Campaigns Experience in Blackbaud Luminate Online®
- 109 EAP for 360 Student Profile in Blackbaud Student Information System
- 41 EAP for Assessment Builder in Blackbaud Learning Management System™
- 9 Technical Preview for SKY API for Blackbaud CRM™ and Blackbaud Altru®
- 55 Community Advisory Group
- 46 Blackbaud Community Ideas
- 26 Blackbaud Community Challenges
- 7 Security Testing Forum
- 1.1K ARCHIVED FORUMS | Inactive and/or Completed EAPs
- 3 Blackbaud Staff Discussions
- 7.7K ARCHIVED FORUM CATEGORY [ID 304]
- 1 Blackbaud Partners Discussions
- 1 Blackbaud Giving Search™
- 35 EAP Student Assignment Details and Assignment Center
- 39 EAP Core - Roles and Tasks
- 59 Blackbaud Community All-Stars Discussions
- 20 Blackbaud Raiser's Edge NXT® Online Giving EAP
- Diocesan Blackbaud Raiser’s Edge NXT® User’s Group
- 2 Blackbaud Consultant’s Community
- 43 End of Term Grade Entry EAP
- 92 EAP for Query in Blackbaud Raiser's Edge NXT®
- 38 Standard Reports for Blackbaud Raiser's Edge NXT® EAP
- 12 Payments Assistant for Blackbaud Financial Edge NXT® EAP
- 6 Ask an All Star (Austen Brown)
- 8 Ask an All-Star Alex Wong (Blackbaud Raiser's Edge NXT®)
- 1 Ask an All-Star Alex Wong (Blackbaud Financial Edge NXT®)
- 6 Ask an All-Star (Christine Robertson)
- 21 Ask an Expert (Anthony Gallo)
- Blackbaud Francophone Group
- 22 Ask an Expert (David Springer)
- 4 Raiser's Edge NXT PowerUp Challenge #1 (Query)
- 6 Ask an All-Star Sunshine Reinken Watson and Carlene Johnson
- 4 Raiser's Edge NXT PowerUp Challenge: Events
- 14 Ask an All-Star (Elizabeth Johnson)
- 7 Ask an Expert (Stephen Churchill)
- 2025 ARCHIVED FORUM POSTS
- 322 ARCHIVED | Financial Edge® Tips and Tricks
- 164 ARCHIVED | Raiser's Edge® Blog
- 300 ARCHIVED | Raiser's Edge® Blog
- 441 ARCHIVED | Blackbaud Altru® Tips and Tricks
- 66 ARCHIVED | Blackbaud NetCommunity™ Blog
- 211 ARCHIVED | Blackbaud Target Analytics® Tips and Tricks
- 47 Blackbaud CRM Higher Ed Product Advisory Group (HE PAG)
- Luminate CRM DC Users Group
- 225 ARCHIVED | Blackbaud eTapestry® Tips and Tricks
- 1 Blackbaud eTapestry® Know How Blog
- 19 Blackbaud CRM Product Advisory Group (BBCRM PAG)
- 1 Blackbaud K-12 Education Solutions™ Blog
- 280 ARCHIVED | Mixed Community Announcements
- 3 ARCHIVED | Blackbaud Corporations™ & Blackbaud Foundations™ Hosting Status
- 1 npEngage
- 24 ARCHIVED | K-12 Announcements
- 15 ARCHIVED | FIMS Host*Net Hosting Status
- 23 ARCHIVED | Blackbaud Outcomes & Online Applications (IGAM) Hosting Status
- 22 ARCHIVED | Blackbaud DonorCentral Hosting Status
- 14 ARCHIVED | Blackbaud Grantmaking™ UK Hosting Status
- 117 ARCHIVED | Blackbaud CRM™ and Blackbaud Internet Solutions™ Announcements
- 50 Blackbaud NetCommunity™ Blog
- 169 ARCHIVED | Blackbaud Grantmaking™ Tips and Tricks
- Advocacy DC Users Group
- 718 Community News
- Blackbaud Altru® Hosting Status
- 104 ARCHIVED | Member Spotlight
- 145 ARCHIVED | Hosting Blog
- 149 JustGiving® from Blackbaud® Blog
- 97 ARCHIVED | bbcon® Blogs
- 19 ARCHIVED | Blackbaud Luminate CRM™ Announcements
- 161 Luminate Advocacy News
- 187 Organizational Best Practices Blog
- 67 everydayhero Blog
- 52 Blackbaud SKY® Reporting Announcements
- 17 ARCHIVED | Blackbaud SKY® Reporting for K-12 Announcements
- 3 Luminate Online Product Advisory Group (LO PAG)
- 81 ARCHIVED | JustGiving® from Blackbaud® Tips and Tricks
- 1 ARCHIVED | K-12 Conference Blog
- Blackbaud Church Management™ Announcements
- ARCHIVED | Blackbaud Award Management™ and Blackbaud Stewardship Management™ Announcements
- 1 Blackbaud Peer-to-Peer Fundraising™, Powered by JustGiving® Blogs
- 39 Tips, Tricks, and Timesavers!
- 56 Blackbaud Church Management™ Resources
- 154 Blackbaud Church Management™ Announcements
- 1 ARCHIVED | Blackbaud Church Management™ Tips and Tricks
- 11 ARCHIVED | Blackbaud Higher Education Solutions™ Announcements
- 7 ARCHIVED | Blackbaud Guided Fundraising™ Blog
- 2 Blackbaud Fundraiser Performance Management™ Blog
- 9 Foundations Events and Content
- 14 ARCHIVED | Blog Posts
- 2 ARCHIVED | Blackbaud FIMS™ Announcement and Tips
- 59 Blackbaud Partner Announcements
- 10 ARCHIVED | Blackbaud Impact Edge™ EAP Blogs
- 1 Community Help Blogs
- Diocesan Blackbaud Raiser’s Edge NXT® Users' Group
- Blackbaud Consultant’s Community
- Blackbaud Francophone Group
- 1 BLOG ARCHIVE CATEGORY
- Blackbaud Community™ Discussions
- 8.3K Blackbaud Luminate Online® & Blackbaud TeamRaiser® Discussions
- 5.7K Jobs Board