[Tex/LaTex] Avoiding duplicate entries in bibliography having different cite keys

bibliographiesbibtexbibtool

I have several manuscripts each with their own bibliography (*.bib file). I'm now trying to combine these manuscripts into one document having one single bib file. My problem is that I have duplicate entries in these bib files that I would like to remove.

My OS is OS X Yosemite, so I have access to BibDesk if that's an option (I mention this because this came up in several Google searches prior to me asking this question). I've also tried to use bibtool to comment out duplicates. It works well, but only on entires that are exact duplicates. My problem is that for the same reference, I have different cite keys across my different bib files.

My question is: is there any way to clean up the merged bib file and either assign two cite keys to one reference—or—automatically detect duplicates (with different cite keys)?

Thanks.

Best Answer

Yes, both BibDesk and Jabref have internal duplicate finder tools. But none of them have advanced ways of cleaning up duplicates. While Jabref is still better in giving the user an option to Merge or select any instance of the duplicates, its algorithm of finding the duplicate is not transparent. It also often fails to detect entries with exactly the same title.

The problem is: both of the reference managers seem to detect duplicates only when the entireshave exactly the same values in all the required fields. I have about 4000 references. Both Jabref and Bibdesk detected a few references at first. After some cleaning up, then, they reported absence of duplicates. But I knew I have more than 500 duplicate entries in my library. I know this because I tested the library in Bookends; a reference manager which has more advanced duplicate finder tool.

To find a permanent solution, I have requested for more advanced duplicate finder tool in Jabref :http://discourse.jabref.org/t/more-control-on-the-duplicate-finder/120 (For the Java programmers, it would be great if you can offer a hand on the Jabref.)

In the meantime, I have asked in this website for a solution based on AppleScript (BibDesk accepts AppleScript). The user jacksr300 helped me with an AppleScript. I am now able to clean all my duplicates based on Title as well as CiteKey. Check out the answer here: https://stackoverflow.com/questions/39427926/find-and-remove-duplicates-in-a-bibtex-bibdesk-using-applescript/39439542?noredirect=1#comment66377804_39439542