[Tex/LaTex] Large bibtex/biblatex database

biblatexbibtexfile size

I'm not sure if it is a good idea to maintain a single large bibtex database or if I should split it into multiple smaller ones. I have compiled a very large India-specific database (over 100,000 records) from various sources.

So, what are the most important positives and negatives for one approach or the other one? Is there any difference in how bibtex and biblatex treat big data files? Are there any threats to be considered with either approach?

Best Answer

Personally, I have two different bib-files; one for publications by others and one for my own publication list. But even this is just a matter of convenience because the entry-groups differ between the two for a more convenient working with the bib-files themselves, not considering any LaTeX compilation.

My main bib-file is well in access of 1000 entries, and I never experienced any problems with any compiler (BibTeX, BibTeX8, BibLaTeX) so far. I have never heard of any instance where such problems occurred either. True, there is a limit for the number of possible bibliography entries (defined by bytes to be processed) in BibTeX, and also in BibTeX8 (altough in the latter you can expand that limit to such large values that I did not run into problems with my Diploma thesis which contained tons of citations). But this has to do with the number of citations in your document and not the file size of the bib-file, so if you run into this problem it would persist even if you would split your bib-file. As far as I know, BibLaTeX does not have such limits anymore at all.

Another thing to consider is how easy it is to merge entries from multiple bib-files into one bibliography. In BibLaTeX, where you can declare several source files easily and create the bibliography by an altogether different command, this is easily done. In BibTeX and BibTeX8 it is to my knowledge not so easy (if possible at all) to compile information from several bib-files into one reference list.

So I would say is it mainly a matter of personal taste, probably also depneding on how much you are using the bib-files directly---it can make sense to have several if the entries are very different in topic. Compilation will work with either version at least in BibLaTeX, but may require additional packages or merging of bib-files for BibTeX/BibTeX8.

Related Solutions

[Tex/LaTex] Statistics about your bibtex database

I don't know of such a tool, but it should be possible to process BibTeX data in statistical software (such as R) or even in Excel.

The bibtex package on CRAN can parse BibTeX files. Converting to a data frame should be possible on an attribute-by-attribute basis. Then you can analyze whatever you want. See also this related question on SO.

> rref <- read.bib()
> rref
R Development Core Team (2009). _R: A Language and Environment for
Statistical Computing_. R Foundation for Statistical Computing, Vienna,
Austria. ISBN 3-900051-07-0, <URL: http://www.R-project.org>.
> str(rref)
List of 1
 $ :Class 'bibentry'  hidden list of 1
  ..$ :List of 7
  .. ..$ title       : chr "R: A Language and Environment for Statistical Computing"
  .. ..$ author      :Class 'person'  hidden list of 1
  .. .. ..$ :List of 5
  .. .. .. ..$ given  : chr "R Development Core Team"
  .. .. .. ..$ family : NULL
  .. .. .. ..$ role   : NULL
  .. .. .. ..$ email  : NULL
  .. .. .. ..$ comment: NULL
  .. ..$ organization: chr "R Foundation for Statistical Computing"
  .. ..$ address     : chr "Vienna, Austria"
  .. ..$ year        : chr "2009"
  .. ..$ note        : chr "{ISBN} 3-900051-07-0"
  .. ..$ url         : chr "http://www.R-project.org"
  .. ..- attr(*, "bibtype")= chr "Manual"
  .. ..- attr(*, "key")= chr "R"
 - attr(*, "class")= chr "bibentry"
 - attr(*, "strings")= Named chr(0) 
  ..- attr(*, "names")= chr(0) 
> rref$url
[1] "http://www.R-project.org"

If you prefer Excel, you might want to convert your bib file to XML first. I haven't tried that, though.

[Tex/LaTex] BibTex style database

It should be more a comment than an answer but I do not have enough rep to comment.

I was managing the bst database at bst.maururu.net but do not have enough time to continue doing so. Of course, I kept a dump of the db, the code of the website and, most importantly, the archive of all styles. To host it, one would need a web server with MySQL and PHP, at their most basic versions.

I would be happy to transfer all that to anyone willing to handle it. Thank you in advance for your help!

Best Answer

Related Solutions

[Tex/LaTex] Statistics about your bibtex database

[Tex/LaTex] BibTex style database

Related Question