Update 2013-11-11: For some statistical data read On the popularity of reference managers, and their rise and fall
Update 2014-01-15: For a detailed review of Docear and other tools, read Comprehensive Comparison of Reference Managers: Mendeley vs. Zotero vs. Docear

At time of writing these lines, there are 31 reference management tools listed on Wikipedia and there are many attempts to identify the best ones, or even the best one (e.g. here, here, here, here, here, here, here, here, … [1]). Typically, reviewers gather a list of features and analyze which reference managers offer most of these features, and hence are the best ones. Unfortunately, each reviewer has its own preferences about which features are important, and so have you: Are many export formats more important than a mobile version? Is it more important to have metadata extraction for PDF files than an import for bibliographic data from academic search engines? Would a thorough manual be more important than free support? How important is a large number of citation styles? Do you need a Search & Replace function? Do you want to create synonyms for term lists (whatever that means)? …?

Let’s face the truth: it’s impossible to determine which of the hundred potential features you really need.

So how can you find the best reference manager? Recently we had an ironic look at the question what the best reference managers are. Today we want to have a more serious analysis, and propose to first identify the bad reference managers, instead of looking for the very best ones. Then, if the bad references managers are found, it should be easier to identify the best one(s) from the few remaining.

What makes a bad – or evil –  reference manager? We believe that there are three no-go ‘features’ that make a reference manager so bad (i.e. so harming in the long run) that you should not use it, even if it possesses all the other features you might need.

1. A “lock-in feature” that prevents you from ever switching to a competitor tool 

A reference manager might offer exactly the features you need, but how about in a few years? Maybe your needs are changing, other reference managers are just becoming better than your current tool, or your boss is telling you that you have to use a specific tool. In this case it is crucial that your current reference manager doesn’t lock you in and allows switching to your new favorite reference managers. Otherwise, you will have a serious problem. You might have had the perfect reference manager for the past one or two years. But then you are bound to the now not-so-perfect tool for the rest of your academic life. To being able to switch to another reference manager, your reference manager should be offering at least one of the following three functions (ideally the first one).

  1. Your data should be stored in a standard format that other reference managers can read
  2. Your reference manager should be able to export your data in a standard format
  3. Your reference manager allows direct access to your data, so other developers can write import filters for it.

The latter point is of particular importance if you think about using a web-based reference manager (e.g. CiteULike or Bibsonomy). In this case, chances are that other reference managers cannot access you data, especially not, if the provider of the web-based reference manager decides to turn-off its service. So, you should definitely check if the web-based reference manager allows you to export all your data or allows accessing the data via an API. We emphasize the “all your data” because some reference managers only offer an export function for some of your data. This means, only because you see an export button in your favorite reference manager, you shouldn’t think “All right, there is an export function, everything is great”.

A prominent example for a non-complete export function is Mendeley. Mendeley allows exporting bibliographic data of your PDFs (including tags) as BibTeX file. However, Mendeley does not export the virtual folder structure in which you organize your PDFs. More importantly, you cannot export text you highlighted in PDFs, or comments you made in your PDFs [2]. Imagine, you spent dozens of dozens of hours sorting your PDFs to the folders in Mendeley, and annotated your PDF files but then, in a few years, you decide to use another tool than Mendeley. You would want to export all your data but couldn’t. So, you would have to make a decision: Do you annotate all your PDFs again with the new reference manager or just stick with Mendeley, even if it’s not the best solution (any more)? That’s a dilemma – you are locked-in.

Another example is Qiqqa and ReadCube. As far as we know, Qiqqa and Readcube store your PDF annotations in a proprietary format which cannot be exported to the standard PDF format. Hence, if you are using Qiqqa or ReadCube (and annotate PDFs) you will be locked-in in a similar way as with Mendeley. [Update 2014-11-26: Please read the comments for more information about Qiqqa]

2. A “must-register” feature that forces you to store your data in the cloud

In the good old days, you downloaded a software, installed it on your computer and your data was all yours. Nowadays, there is the cloud. We won’t discuss here whether storing data in the cloud is generally sensible or not. However, we argue that you shouldn’t use a reference manager that forces you to store your data in the cloud, even if you personally advocate the cloud. Fact is that there is a growing concern about privacy issues relating to cloud storage. As a consequence, some companies ban cloud services from their employees’ computers. For instance, IBM banned services such as Dropbox because of privacy concerns, and from personal experience we know that also some universities are quite skeptical about Dropbox & Co. When you use a reference manager, that stores all your data in the cloud, we see a good chance that someday your university will ban this tool, because the university does not want their employees’ highly sensitive data to be stored in the cloud. We don’t know yet of any universities who did so (do you?), but at least in smaller scale this already happens. For instance, I recently met a researcher at the JCDL conference. He just got a PostDoc position and his Professor told him, he could use any reference manager he wanted, as long as it doesn’t store his and the working group’s data in the cloud. Actually, the researcher was using Mendeley and although he was totally happy with it, he was forced to use another tool, only because Mendeley had no option to work locally, without registration. So, at least if you are considering a long term career in academia, we suggest to use a reference manager that does not force you to store your data in the cloud. Otherwise, you might be forced to migrate to another tool due to to some policies of your boss or university (and migrating is usually very time consuming, or, if there is a lock-in, not possible at all).

3. A “loner feature” that prevents you from collaborating with your colleagues

Sooner or later you will be required to collaborate with other researchers, at least occasionally. Then, your reference manager hopefully will support this or at least it will not prevent you from doing so.  There are three factors that could make a collaboration difficult or even impossible.

A) Proprietary data-formats

In the ideal case your collaborator is just using the same tools as you are – but what if not? Then, it becomes crucial that the reference manager of your collaborator can read the data of your tools and vice versa. That means, when selecting your favorite reference manager, you should make sure it can read and write standard data formats and is not using proprietary formats. Mendeley, for instance, will make it difficult (if not to say impossible) to collaborate with researchers using a reference manager other than Mendeley. As explained earlier, Mendeley has no export (or import) for annotations in PDF files. So, you cannot share your annotations with colleagues when they are using PDF editors other than that of Mendeley. Also sharing and working on the same BibTeX file would be difficult. Although Mendeley can import and export BibTeX files, Mendeley does not update existing entries in its database with entries that have been changed in the BibTeX file. This means, if you export Mendeley’s bibliographic data as BibTeX, and your colleague is changing some data in the BibTeX file, you may import the BibTeX file again, but all modified documents are imported as new documents, so you would have duplicate entries in Mendeley.

B) Infeasibility for others to use your favorite reference manager

If the data format of your reference manager is not compatible with the tools of your collaborators, you could still try to make your collaborators using the same software as you are doing. However, even if they wanted, there might be reasons preventing the use of your favorite reference manager.

First, not all tools are available for all operating systems. If you are using a reference manager for Windows, but your collaborator is using Linux or Mac then you have a serious problem. Of course, the collaborator could try to run the reference manager in a virtual machine, or install Windows but this is a very error prone and time consuming task (and costly, if the collaborator has to buy a Windows license).  Most likely, not all your potential collaborators would be willing to do so. This means if you decide to use a reference manager not available for all operating systems, you most likely will have to compromise in the choice of your collaborators.

Second, the price of the software might be important. A tool like Endnote costs about 250$ for a single standard license. Even if you think the software is worth it, or if you get the software for free through your university, not all potential collaborators can or want to pay the price (think of students, researchers in developing countries, or researchers who just want to have a quick look at your data).

c) Local databases

If you are using a reference manager that stores your data e.g. in BibTeX format, collaboration is easy. You can just store the BibTeX file in your Dropbox and share it with collaborators. Alternatively, you could setup e.g. SVN which would not only sync BibTeX files but merges them when several users make changes at the same time. Similarly, you could share PDFs via Dropbox and annotate them with standard PDF readers such as Adobe Acrobat, Foxit Reader, … However, many recent reference managers store their data not in individual files (e.g. BibTeX, or PDFs) but in a local database. Databases have several advantages for your daily work but they have one huge disadvantage: You cannot simply copy them to your Dropbox and share them with your colleagues. If these reference managers do not offer some own functionality for collaboration, it will be almost impossible for you to share your data. If the reference manager offers an integrated collaboration function this might even be more comfortable and powerful than e.g. Dropbox. However, it usually comes at a price and that is typically quite high. In case you don’t want to collaborate but to synchronize your data between e.g. your home and office computer, your reference manager should at least allow to change the path of the local database. In this case you can store your database e.g. in Dropbox and use it on another computer.

Therefore, if you look for a reference management tool, check if you can use standard synchronization tools such as Dropbox (or other third party solutions). If you cannot, look at the pricing plans and really think about if you are willing to pay the price. For instance, at time of writing, Mendeley charges 99€ (~130$) a month for their  business team package with five users. We are not saying that Mendeley’s service is not worth the money. Actually, their collaboration features are great and far more powerful than e.g. Dropbox. But 99€ a month (i.e. ~1200€ a year!) is still quite a bit of money for five users to collaborate. Think about, if you are really willing to pay this price and whether potential collaborators of yours would be willing to pay their share, too.

Summary & Conclusion

Many researchers are selecting their favorite reference managers based on what wonderful features the tools offer. However, we argued that in a first step researchers should think about what ‘features’ the reference manager should not offer. We proposed that reference managers that a) lock you in, b) force you to store your data in the cloud, or c) prevent collaboration, should be ignored right away. Maybe, if you are a Bachelor or Master student, and you are 100% sure that after handing in your thesis, you will never work again with academic literature, you might ignore the advice we proposed. However, if you are planing to work for several years in academia (or if you are uncertain), we suggest to think about what we wrote. Otherwise, chances are that you won’t be happy with your reference manager in the long-run.

It’s probably needless to say that Docear is none of the bad reference managers. Where ever possible we use open standard formats to store your data and all your data is on your own computer and (optionally) on our servers*. Your PDF annotations are stored in the PDF standard format, and any standard PDF viewer will be able to read them, and you can share your PDFs (and annotations) easily e.g. via Dropbox. Your references are stored in BibTeX format which can be read by almost any reference manager and sharing and collaborating with BibTeX files is easy. Mind maps are stored in Freeplane’s XML format, a format that is understood by several other tools and that any developer could easily write an importer for within hours. You may export your mind maps in various other formats (e.g. PDF, PNG, …) and, again, you can share them easily. Even the source code of Docear is publicly available. If we should ever stop developing Docear, anybody could continue our work.

Does all this mean that Docear is the best reference manager? Not necessarily. Only because something is not bad, doesn’t make it automatically good. However, we hope that our argument helps to avoid selecting a reference manager that will make you regret your decision, and we are very keen to hear which reference managers you think belong to the bad category and which don’t.  How about BibDesk? Bibsonomy? Bibus? Citavi? CiteULike? EndNote? Mendeley? Papers? Qiqqa? Refbase? RefWorks? Zotero? …? We are also looking forward to hearing any other of your ideas about this topic. Did we forget some no-go features? Do you agree with our list? Are we exaggerating? Were we too biased when writing this article? Let us know!

Update: 2013-11-20: I created a table showing the three ‘negative-features’ for some selected reference managers. Please help us to complete the table. If you know a missing piece, send the information to info@docear.org. Also, if you discover errors, please let us know!


[2] In Mendeley, you can export each of your PDFs individually and your comments will be added to the last page of the PDF. However, this is hardly a true export of your data and no other tool will be able to recognize the annotations in the PDF as real annotations but only as normal text. This also means that if you export Mendeley’s annotations, you might see them in your favorite PDF viewer but you won’t be able to edit or delete the annotations. For more information on the shortcomings of Mendeley’s PDF export, read the comments.

* Currently, we only backup your mind maps, but we are working on a backup for PDFs and reference data.

Update note, November 1st 2013: The “must-registration” feature was added.


Joeran Beel

Please visit https://isg.beel.org/people/joeran-beel/ for more details about me.

36 Comments

Ivan Arnold · 21st December 2018 at 15:41

I think your assessment of Mendeley is a little harsh. Specifically, some of the points you make in your ‘three negative features’ table concerning Mendeley are simply not correct.

Mendeley might not export bibliographies in PDF format, but I really don’t see why that is particularly useful. It does, however, export standard bibtex and offer a Word and LibreOffice plugin. Since this should cover basically everyone, and all of these export bibliographies to PDF, I don’t see your point.

Second, Mendeley DOES NOT require that you ONLY store data in the cloud. True, it does back up your data to the cloud, though I really don’t see how this could be considered a bad thing, unless privacy concerns extend to a collection of academic papers. However, Mendeley also allows you to store PDF’s and a bibtex backup locally, which it periodically updates. I find this feature useful.

Finally, Mendeley allows for fairly easy collaboration, though you are limited to three (I think) collaborators unless you have a pro account.

I’m not saying that Mendeley is the best reference manager. I use it, and it makes my life easier. Also, it works on all platforms (Linux, Mac, Windows) and that’s nice for me.

Thanks, just my two cents.

    Matthew Fero · 18th December 2019 at 11:59

    Ibid for ReadCube. It syncs to cloud, but I keep full libraries on my workstations. It exports in Bibtex, EndNote, and RefMan formats, including the user’s “Notes” field, but not PDF highlights and annotations.

Keinsein Junior · 2nd October 2018 at 09:51

Bibsonomy people are open for Desktop usage. They offered a JabRef plugin for several years and allow to work with a public (synchronisation) API. Unfortunately JabRef stopped to support plugins. The server parts are or were open Source so that you can set up your own bibsonomy network. Collaborative work is possible by creating groups.
One difference between Bibsonomy and other platforms is that they have a scientific background. They focus more on algorithms that work with the database than on the data itself (thogh they need them for playing withh their algorithms). This meens that the some features try to be more intelligent than other tools provide, while they may not meet the needs of all people.

James · 14th June 2018 at 14:26

Joeran is absolutely right. I just killed Mendeley on my computer due to pdf annotations issue. Now I use zotero to manage my resources and Docear to write literature review.

Chris Kent · 16th December 2014 at 04:22

Hi

Interesting topic and many comments I have read with interest as a Master’s student who has recently moved from Mendeley to Qiqqa. I found Mendeley a great REFERENCE Manager and still keep it ‘just in case’ but as my studies have progressed I discovered Qiqqa which takes things to the next level as both a Reference and RESEARCH management tool. One thing that concerned me was being ‘tied in’ and having my data inaccessible in ‘every day format’.

I read James’ post dated 25th Nov ’14 and have tested this for myself. I did have some concerns with Qiqqa data management being buried in the depths of an inaccessible database so I placed my PDF’s into a designated directory before loading them into the Qiqqa database to make sure I had a copy. Mendeley likes to use a designated directory so this wasn’t a problem. And I like to take a good old fashioned backup of a directory so I know I’m safe.

Tonight I can rest assured now that I have tested the Qiqqa Export facility which nicely drops a copy of the original PDF and annotated PDF into separate directories (see James’ post for details) and ‘Voila’ the sun shines!

WHAT this does highlight is the issue of User Data Management and after 25+ years in IT it is a subject I am familiar with.

If the ownership and accessibility of source files concerns you then the following procedure may help:
1) Download the article PDF into a dedicated directory on your hard drive e.g. ‘C:xxxLiterature Review’
2) Load this file into Qiqqa
3) Annotate as required
4) back up ‘C:xxxLiterature Review’ as with any other valuable data

and you can also
5) Export the data as per James’ article 25th Nov
6) Back up BOTH original and annotated PDF to other devices

My conclusion is that neither Qiqqa or Mendeley ‘Lock you in’ from a data perspective as you can access your data in its original PDF format very easily (and edited format). Therefore if I, for example, change jobs and am forced to to use another system I merely export from Qiqqa or copy my source directory and import all my PDF’s to the new library and “Hey Presto!” job done!

I have not tested how annotations behave but certainly for source data there is not an issue and you are free to roam the referencing wilderness as you see fit taking your bag of PDF’s as required…

Certainly for Mendeley & Qiqqa the comments in the ‘Lock In’ column need amending and with a little careful (user) data management there is no problem to solve.

Chris

James Phillips · 1st December 2014 at 11:08

Thank you for those corrections Joeran!

Yes, the most important information from the users’ Qiqqa annotation, their comment and tags, are in the editable PDF sticky note. The box highlight is a standard PDF element and requires a PDF editor to delete; it’s the only way to export Qiqqa’s advanced annotations for standard PDFs. We like users of Qiqqa owning all their data, having it easily exportable, and knowing they’re not locked in.

Best,
James & the Qiqqa team

James Phillips · 25th November 2014 at 18:42

Another example is Qiqqa and ReadCube. As far as we know, both tools store your PDF annotations in a proprietary format which cannot be exported to the standard PDF format. Hence, if you are using Qiqqa or ReadCube (and annotate PDFs) you will be locked-in in a similar way as with Mendeley.

Hi, we’re proud to say that you have always owned your data with Qiqqa: you will always be able to freely export it to standard formats and have been able to for many years: http://blog.qiqqa.com/post/103560922552/you-own-your-data

Also in the table it says you need to register with Qiqqa and infers all your data is always stored in the Qiqqa cloud. Neither of these have been true since the beginning of Qiqqa. You can download and install the Qiqqa client and use it with Guest access without needing to sign up. Also in Guest mode your data will only be on your computer and will never be sync’d to the Qiqqa cloud. To sync to the cloud, you’d need to register and press Sync in the Qiqqa client. @Joeran, it’d be great to see the article edited to correct those mistakes. I have dropped you an email at info@docear.org too just in case you don’t see this comment. Thanks.

Also I’d be more than happy to chat any time you want to clarify what Qiqqa can and can’t do. Glad to see the issue of freedom of data and lock-in being raised and getting the attention it deserves.

Thanks,

James & the Qiqqa Team

    Joeran [Docear] · 26th November 2014 at 10:07

    Hi James, you are right, Qiqqa does export annotations. I am sorry for the mistake, and updated my original post and the table. However, it appears to me that Qiqqa exports highlighted text in a way that makes it impossible to delete or modify this highlighted text with a standard PDF viewer. And the comments seem to be “enriched” with a color box in the background that cannot be deleted (the exported comments in the PDF can be deleted or modified; but not the box). Is this correct?

    Btw. we didn’t get your email.

Chris twomey · 6th November 2014 at 02:02

I would echo the points made by Docear here; non-standard storage of PDF highlighting and annotations is a major way to lock in people. I am increasingly frustrated with Sente for many reasons, but will struggle to jump ship because of this. So to Mr. Gunn’s comment that exporting annotations is solely an issue for sharing with other users, well, that assumes that your users always want to be your users. That is the whole point of locking them in. If you want to give them the freedom to easily leave, then the issue is much broader.

Kuddos for the Docear project. I may not jump into it (reading on iPad is key for me), but certainly plan to monitor its development over time. Looks very promising at this point.

Denis · 27th June 2014 at 17:57

thanks for a nice write up. I really enjoyed reading.

I am a Papers (2 and 3 ) user for quite a while. But I really hate the fact that they become more and more lock-in.
Luckily, they still support proper BibTex output with links to .pdf’s.
But, annotation is done within the app!

I think will be switching to JabRef or Docear, which have less features but are much more open…

Joeran [Docear] · 20th January 2014 at 16:46

One user send this by email, so I thought I share it here:

Hello

Some added information:

Regarding you table on
https://ISG.beel.org/blog/2013/10/14/what-makes-a-bad-reference-manager/
Citavi has a “loner feature”! It runs only on Windows, so there is a problem of sharing with colleagues who use Linux, Mac etc.

Best
***

Jesse Engel · 8th January 2014 at 15:34

I just want to echo what David Boutelier said. Mendeley has worked well for me until my current postdoc where I need to sync 3 different computers with 3 different operating systems.

David Boutelier · 16th December 2013 at 20:29

Interesting post. Thanks. I loved Mendeley until I tried to move my library from a Mac to PC when relocating. They want you to use their cloud solution to synchronize everything. That’s a great solution if you only have a few pdfs. But over the year I piled over 5GB of pdfs and I don’t want to pay their premium services required to synchronize large datasets just for a move. So I moved my database and folders manually but somehow screwed up the database in the process. Then I discovered that all the reference information is in the database. So my new Mendeley was extracting the reference information for my old papers from the web again… Nothing is stored in the pdf. I since switched to JabRef. If I decide to switch over again, my reference data are stored in the pdfs. As long as the new reference manager I use can import from xmp metadata I am fine. But that is not the case of Mendeley or Readcube. Anybody knows if Zotero reads xmp?

Cheers,

granger · 12th December 2013 at 00:53

hi Joeran,

great and convincing post, now I know which ref managers should I try! ..may I suggest a fourth column, titled “Freedom”, for the table?
nonfree, proprietary software makes you depend entirely on the developer, and that is the ultimate lock-in. it is bad, unethical, evil! 😀
http://www.gnu.org/philosophy/proprietary.html

but free software (like docear) gives the user the freedom to share, study and modify it.
🙂

regards

Jonas Scheel · 21st October 2013 at 13:10

Quiz:

Is Mendeley’s proprietary annotation export a lock-in?

[ ] “There is absolutely no lock-in with Mendeley whatsoever” (Mr. Gunn, Mendeley)
[ ] Not really
[ ] Maybe a little bit
[ ] Yea, quite a bit
[ ] Yes, pretty much
[X] Yes! Yes! Yes!

Is Docear’s concept just extraordinary?
[X] Yes
[X] Yes
[X] Yes
[X] Yes

Is Mendeley’s PDF meta data extraction better than that of Docear’s?
[ ] No
[ ] A little bit
[ ] Quite a bit
[X] Yes! Yes! Yes!

Is Mendeley just damn easy to use (compared to Docear)?
[X] Yes
[X] Yes
[X] Yes
[X] Yes

Should I use Docear or Mendeley?
[ ] Docear
[ ] Mendeley
[X] Dammit, I don’t know. Why is there no Mendel-ear? 🙂

🙂

Andrew King · 18th October 2013 at 18:55

I’ve used Mendeley far more than I’ve used Docear (and don’t use either at the minute!) and I’d have to say that Joeran gave a fair assessment of where Mendeley is at. Although note it is easy to export your references on a per virtual folder basis just go into the Bibtex settings and set as per folder rather than whole library basis.

I loved using Mendeley and would have been happy to pay except for the following two issues:

1)Annotations could only be exported on a per pdf basis and even then only as ‘proxy’ annotations all piled up at the end.

This is a big deal, I have a totally electronic workflow if I can’t get ‘as shown’ access in a standards compliant pdf reader to the annotations I’ve invested time in making then I’m just not happy using the tool.

Not only that but I’d have lost access to any of the many great tools for working with notations in standard pdf’s eg itext in java.

Oh and for the vast majority of researchers saying access is available via the API is equivalent to saying that there is no access. It’s hardly the difference between different file formats, it requires a whole different skill set to use. If I understand MrGunn’s role at Mendeley correctly I can see why in his world it would be a no brainer but in the wider community you would just get blank looks.

2)The mobile app appeared and then just seemed to be left in a poor state. I thought this was going to be the killer one-two from Mendeley a great desktop app with on the go access from a company built on the web.
Unfortunately that never happened I get a far superior user experience from Refmaster or Pocketbib. Which is frankly bizarre given they are based on ye olde bibtex and I think are both single developer projects.

Well that was a lot about Mendeley on a Docear blog!
I guess the point that’s relevant if you use Docear is free your bibtex it works well on iOS and Android.

Congratulations on the 1.0 release Joeran. I’m off to take Docear for another spin and see how its improved.
Andrew

—————————–
I don’t know if Mr. Gunn will return to the blog but if you do and are interested in feedback from an ex-user who wanted to stay with your project you can get hold of me @nopapyrus . In turn I’d be intrigued to learn about the reasoning behind some of the calls made on what was a priority in developing Mendeley.

Mr. Gunn · 18th October 2013 at 01:26

OK, so I hear your feedback about the PDF annotations. We are aiming for maximum interoperability and that’s why we’re keeping an eye on the Open Annotation W3C standard and plan to be compatible with it.

We believe in making products people want to use because they simply work better, not because the software locks the user in.

Joeran [Docear] · 16th October 2013 at 15:38

I would like to quote a tweet from Twitter that a user (Aurimas) sent:

#refworks (doesn’t let you export PDFs in open format) and #noodletools (no export at all) are really bad.

Can someone confirm that RefWorks doesn’t let you export PDFs in open formats, and that NoodleTools has no export at all?

    adam.smith · 16th October 2013 at 19:46

    Yes, that’s correct. Aurimas has worked quite a bit on this.

    RefWorks data exports (including it’s own RW Tagged and RW XML formats) don’t include folder information and no attachments). It is possible to download a RW library for back-up, but that is a password protected zip file (it’s not hard to hack that, but it’s certainly not intended to be and is at best a legal grey area.)

    Last time we looked at this for NoodleTools was about a year ago and the user we tried to help confirmed this with their support.

Piotr · 16th October 2013 at 10:12

My bad, Mendeley has indeed two more export formats. Somewhat hidden in the menu, though.

As Joeran said, my turn from using Mendeley (and Zotero) is driven by the superior annotation system, which is provided to be honest rather by PDFXV then Docear 😉 What adds the value to Docear is the mind mapping and automated annotation extraction into the map. It allows to draw connections between papers, between papers and my work and document the work itself (Zoreto allows cross-linking of papers, but cannot visualise the network, nor place annotations on it). With a bit of work Docear can be not only a literature suit, but a research documentation tool. One thing missing is a build-in app for simple project managing (just the Gantt chart) and some sort of calendar allowing making notes in it.

I must also say that I value freedom in using software. Mendeley seems to obligatory bind the user to its cloud and it’s database model (API is useless for the average user). I don’t say it’s ethically wrong, it is OK. Just a business model. But as a consumer I’ve been presented a choice and picked the competitor which has more ‘friendly’ business model.

Mr. Gunn · 15th October 2013 at 20:10

A) Mendeley doesn’t export annotations in bulk, nor in the PDF standard format, but in some awkward proprietary format that does not allow to delete or edit annotations, or to see them where they were created but on the very last page. Hence, users cannot really use their annotations in other tools than Mendeley.

Your main issue here seems to be around annotations, and I’ll concede that we could support them better. That said, I think you went a bit out of your way to make that point. The issue with annotations is the collaborative use case. Writing annotations into the file itself only works for solo use, not for the collaborative use for which we designed it, and annotations will be coming to the API so any client, including yours, can read and write them without mucking around with the PDF itself.

B) You cannot share your bibliographic data and PDFs with researchers who are not using Mendeley, and work on this data collaboratively. This means, if you want to collaborate, you are forced to buy Mendeley’s premium services, and pay the price whatever it is (or you don’t collaborate).

I think you’re making the mistake here, as in your original post, of using the word collaboration to mean something much more narrow. The use case you mention of syncing to and from Bibtex would be a hack that only supports a limited set of our users. There are a range of import and export options in Mendeley, which a quick glance at our menus will show.

C) Mendeley does not export the virtual folder structure in which users organize the PDFs.

The API is the means by which we expose this information. If you don’t like it, that’s your choice, but don’t say we don’t have export when you really mean we don’t have export in the means you would prefer. We do participate in discussions around annotation standards which are compatible with the entire web, not just PDFs, and are in touch with Rob Sanderson and the Hypothesis Project http://hypothes.is in this regard. I’m happy to continue this discussion and would love to see you at the next annotation standards meeting.

    Markus · 15th October 2013 at 20:30

    i’m an ex-mendeley user and think the argument of docear is right. the reason why i switched to docear is that i can use adobe for note taking. and its true what docear says. the pdf export of mendeley is not worth its name. it is no real export. i had to make all my notes again when i switched from mendeley to docear. but i can understand that. why should a company help its users to go away? they must earn money.

    moreover, mendeley forces me to transfer all my data to their cloud. i think that also makes a bad reference manager. in docear i can turn this off.

    Joeran [Docear] · 15th October 2013 at 21:49

    Yes, my main issue is indeed the improper PDF export function. The other issues are just secondary. However, at least for Docear users this issue is really important. Anyway, I highly appreciate your participation in the discussion and I am confident that the readers should now be able to make up their own mind. And I would like to make one point crystal clear: The reason why Mendeley was the only reference manager that I specifically mentioned in my article, was not because I would consider Mendeley to be one of the worst tools ever. The reason was simply that Mendeley is – besides Zotero and JabRef – the tool that I know best and that most of our users (have) use(d). Actually, if Docear wouldn’t exist, Mendeley (or Zotero) would probably be the reference manager of my choice – if Mendeley had a proper PDF export function (and would allow me to collaborate via Dropbox) 🙂

Piotr · 15th October 2013 at 11:25

Hi,

I’m using Mendeley for a few months. I dumped Zotero for Mendely because of the annotations (Zotero still has a better way of fetching stuff on-line, better then Docear too). Can you specify the way or link a webpage with tutorial how can I sent a PDF file annotated in Mendeley to a user not using Mendeley? I started working with some refusing to install “evil” software, whatever that means. With Docear + PDF Xchange Viewer it is very easy. Besides, you must admit Mendeley annotation system is quite limited comparing to PDFXV. And sometimes Mendeley’s highlighting is misbehaving when the PDF has some fancy formatting.

Mendely CAN import BiTeX, EndNode, Zotero formats. But exports only BibTeX. Obviously it’s made to switch to Mendely from other software and BibTeX export is simply a must-have because of all the LaTeX-orientated academic writers. For comparison, Zotero has 14 export formats! (not all equally useful, though) I’m sorry, but that is a lock-in. Maybe not de iure, but de facto! 😛

BibTeX was made to work for LaTeX and does that well. But as a database for working with texts themselves is rather limited. Docear’s mind maps are the extra information layer which makes BibTeX useful again. But still Zotero’s or Mendeley’s database (sqlite in Zotero) gives advantages in terms of searching our data set (that’s why I haven’t dumped Mendeley… yet).

What concerns me in Mendeley is its obligatory on-line service. It’s impossible to register without the on-line account. Mendeley’s syncing between machines is cool, but what if one day my institution will ask us not to use third party on-line services for our work (with a bit of an effort we could have our own servers with literature repositories). I know institutions in my filed which banned using Google Drive for work purposes. Software which can work off-line just fine, like Docear or Zotero (you miss some features, but they will do), will be preferred then.

Piotr

    adam.smith · 15th October 2013 at 18:37

    I think the lack of a proper export function (i.e. one that exports all info & folders) in Mendeley is frustrating (and no, I don’t think the API is a substitute), but in fairness, they do support not just BibTex, but also RIS and Endnote XML.

adam.smith · 14th October 2013 at 22:42

I share some of your concerns, but treating bibtex – which is no standard at all, doesn’t support folders, only supports file attachments with non-standard hacks, and has about a gazillion different flavors – as some type of panacea is odd. I understand bibtex’s appeal, but I find working with it as a data format incredibly frustrating.

    Joeran [Docear] · 15th October 2013 at 10:58

    i agree that working with bibtex can be very frustrating. however, do you know any serious alternative that is used by any application in practice? btw. at least JabRef is storing virtual folders in the BibTeX file. It might a another “flavor” of BibTeX but it works quite well.

      adam.smith · 15th October 2013 at 18:36

      I’m aware of the JabRef virtual folders – we have just added partial support for those in Zotero – but as you say, this is yet another ad-hoc hack. And using Bibtex as a database format just seems too frought with issues. Sure you _can_ modify the file with different software, but how are you ever going to assure that different software does the same thing with a bibtex file, producing all types of chaos as you go back and forth. You’re also forcing people who want to collaborate on a database remotely to work through VCS (if they want to deal with conflicts at all – they can also use Dropbox and just overwrite each others changes…), which sure, is open and all, but not exactly user-friendly. I’ve been using git for years and I still want to shoot myself everytime I have to manually resolve a merge conflict, and dealing with a huge data file that will happen constantly. So basically I think you’re vastly understating the disadvantages of building your entire data model on bibtex, while overselling its advantages. (FWIW, neither Mendeley nor Zotero use proprietary data formats in the strict sense of the term. Both store their database as .sqlite – Mendeley actually does read out Zotero’s sqlite database for one-click import, so this isn’t just a theoretical point. Haven’t looked myself, but others tell me that Mendeley’s sqlite is so intransparent that reading it out in a similar way would be very difficult)

      I don’t think there is a great alternative currently. For collaboration in writing, CSL-JSON may be one option as more and more ref managers use CSL styles. E.g. in theory, though not in practice yet, it would be pretty easy to collaborate on a document between a Zotero and a Mendeley user, as both store citation data in the same format and use the same processor to turn it into citations. I haven’t looked at your Word plugin code, but I’d assume the same would be true for Docear. Cooperation on data curation across reference managers is a thorny issue, because you need to reconcile data models. The most fruitful way forward would likely be to use APIs and an go-between client. E.g. the PaperShip iOS app already syncs both Zotero and Mendeley data individually – next step would be to create an app to allow to sync both _with_ each other.

        Joeran [Docear] · 15th October 2013 at 22:10

        You have a point here. Using databases and collaboration services directly from the reference managers can save you lots of headaches. And it’s true that a sqlite database is not really a proprietary format. But while almost anyone knows how to open and change an XML or BibTeX file in a text editor, the required knowledge for reading data from a database is much higher. Anyway, maybe I should re-think the second point in my article. Even if a reference manager has a “loner feature”, it doesn’t matter as long as you are not locked-in. If you realize some day that you want to collaborate via Dropbox, and your current tool does not allow this, you could still switch to another software. Hm… 🙂

          Alan Belle · 7th January 2014 at 02:54

          Thank you for the interesting posts especially from Mendeley and Zotero and for this thread Joeran.
          Having attempted, with apparent success, to migrate a large EndNote database to Docear (as a hacker not researching bibliographic databases) I agree with adam.smith October 14, 2013 at 10:42 pm “BibTex … is no standard at all” which made mapping difficult. It became clear BibTex is an inconsistent LaTex referencing format, that has been extended to database duties. The benefits of using JabRef and the philosophy of Docear still hold, which is why I am using Docear instead of the University provided EndNote, and request Docear maintain its objective of avoiding a “loaner feature” that would make ease of use simpler to provide, but in the end compromise the whole open philosophical approach.

          My major issue was finding out which flavour of JabRef/Docear understands. Sorry if it is now readily available, or I missed the relevant source at the time.

          PPS: Dorean, I think you could list under “Lock-in” for EndNote as “Complete DB export to BibTex is not possible for the typical user”. The export facility provided in EndNote is quite good (buried in the menus), however they only provide a “citation” export to BibTex. What other fields would they map to and why? They have no commercial reason, unless potential users read Dorean’s caveats.

          If “BibTexDB” as used by JabRef/Docear was defined in the public domain it would address this issue, and critics like myself. It will never be a “true” database, but for most target research databases it is workable with the advantages Dorean identifies.
          PS: Before anyone asks, “BibTexDB” is my name indicating the extra field definitions (like I had to invent) needed if all entries in other databases are to be imported. Given JabRef ignores these fields (except for showing in the “BibTex Source window” it hasn’t cause me incompatibility problems, yet:-). I have not lost the information as it is in a file I can read:-)

        Judson · 11th October 2015 at 23:22

        If you want to shoot yourself every time you have to solve a merge conflict, you need a better tool. Have you used a properly configured visual 3 way diff tool, such as kdiff3?

Mr. Gunn · 14th October 2013 at 21:54

There is absolutely no lock-in with Mendeley whatsoever. In fact, we make all a user’s information available via the API, as well as bulk export in a variety of formats. I don’t understand your point about folder structure. Each user decides on the folder structure they want and we help them organize documents, but the files stay on their disk in the structure they have chosen. They can even have different folder structures on different computers, though I would argue the most efficient way to handle it is to have one big folder and just use tags to filter and search.

You are also incorrect on the following points:
“Mendeley, for instance, will make it difficult (if not to say impossible) to collaborate with researchers using a reference manager other than Mendeley.”
In fact, we have put extensive resources into developing CSL as an open standard, enabling anyone to add citation formatting to any application. See http://csl.mendeley.com/ Did you just miss this or did you mean something else?

“As explained earlier, Mendeley has no export (or import) for annotations in PDF files.”
In fact, Mendeley imports annotations in XMP metadata of PDFs, which is currently the only recognized standard for annotations. You can read annotations made in a variety of PDF readers within Mendeley. As you note in a footnote, you can export annotations written into the PDF, and in addition, we’ve just done the work to allow annotations to sync via our iOS apps, which means annotations are coming to the public API as well.

“In addition, Mendeley can export bibliographic data to BibTeX but cannot import it.”
Mendeley imports Bibtex quite well, actually. Not sure how you missed this bit, but please download our free client and give it a shot. I think you’ll find it works well.

At Mendeley, when I write a blog post mentioning a competitor, I always think twice or three times about it, and then get a neutral friend to read over it, just to make sure it doesn’t seem like I’m unfairly slamming the competition. I think it has improved my writing and also keeps me on friendly terms with those in my own industry.

Cheers from the Mendeley team.

    Joeran [Docear] · 15th October 2013 at 11:34

    Hello William

    thank you very much for taking the time to comment on my post. I am sorry, if I should have treated Meneley unfairly because it was not my intention to make false accusations. Let me clarify some points:

    I don’t understand your point about folder structure. Each user decides on the
    folder structure they want and we help them organize documents, but the
    files stay on their disk in the structure they have chosen.

    This is a misunderstanding. What I mean are the “virtual” folders directly in Mendeley, which users can create in the left window in “My Library”. Please correct me if I am wrong, but I couldn’t find a way to export these folders. This means, when I organize my PDFs in these virtual folders, and decide to switch to another reference manager, I cannot take my virtual folders with me, and would have to re-organize my PDFs in the new software.

    “As explained earlier, Mendeley has no export (or import) for annotations in PDF files.”
    In fact, Mendeley imports annotations in XMP metadata of PDFs, which is currently the only
    recognized standard for annotations.

    As you note in a footnote, you can export annotations written into the PDF,

    I am not sure if we are talking about the same type of “annotation”. When I refer to “annotation”, I mean highlighted text, bookmarks or comments (in Mendeley this is called a “note”). The only recognized standard for annotations in PDFs is the PDF standard (XMP is a standard for metadata such as authors, year, …). As far as I know Mendeley does not store annotations in the PDF standard (neither by default, nor when exporting them). If you create annotations with a normal PDF viewer (e.g. Adobe Acrobat, Foxit Reader, Nitro Reader, PDF XChange Viewer, …), you will not be able to use these annotations in Mendeley. Or, to be precise: You can see highlighted text created with the normal PDF viewer but you cannot edit it, and you can see that there is a comment but you can’t read/edit it in Mendeley.

    Similarly, if you create annotations with Mendeley, you won’t see these annotations in any normal PDF viewer, unless you explicitly export them. However, I consider the export function as useless. First of all, there is no bulk export for PDFs. Second, and far more importantly, the way of exporting annotations is just… awkward. When you export comments (i.e. notes), Mendeley creates a little icon where the comment was originally created. However, the real comments are all on the very last page of the PDF stored as normal text. This means, with a normal PDF viewer, you see that there is a Mendeley-comment e.g. on page 2, and then you need to go to the very last page to see the content of that comment. In addition, you can neither edit nor delete the annotations exported by Mendeley. This means, once you have exported your annotations with Mendeley, they are in the PDF forever. I can really not see why you did the export this way instead of just writing the annotations as normal annotations to the PDF in the PDF standard format. This would show the annotations where they belong, and it would allow to edit and delete them.

    If a Mendeley user spends hundreds of hours reading and annotating his PDFs, and one day he might want to switch to another software, he will realize that he cannot really use his annotations in the new software. Ok, the annotations are there, but all on the last page, they cannot be edited, they cannot be deleted, and they are not recognized as annotations by the new software but only as some text. What else is this than a lock-in? Btw. I am not the only one seeing it like this. Have a look at Mendeley’s feedback forum. Storing annotations in the PDF standard format, is one of your users’ most popular wishes (since years), and some users also consider the status quo as lock in (e.g. first comment, first URL).

    http://feedback.mendeley.com/forums/4941-general/suggestions/1186641-bulk-export-pdf
    http://feedback.mendeley.com/forums/4941-general/suggestions/985703-pdf-annotations-should-be-compatible-with-other-pd
    http://feedback.mendeley.com/forums/4941-general/suggestions/4304446-enable-the-built-in-pdf-reader-to-recognize-annota
    http://feedback.mendeley.com/forums/4941-general/suggestions/1603355-option-to-save-my-highlights-and-comments-in-the-p

    In fact, we make all a user’s information available via the API,
    as well as bulk export in a variety of formats.

    Maybe I am looking at the wrong place, but in the API I can only see methods for retrieving document details such as authors, tags, and citations. I cannot see any option to retrieve annotations. http://apidocs.mendeley.com/home/user-specific-methods/user-library-document-details

    in addition, we’ve just done the work to allow annotations to sync via our iOS apps, which means
    annotations are coming to the public API as well.

    This is great news, and maybe someone will use the API to write a tool that fetches Mendeley-annotations and stores them as real PDF annotations in the PDFs. However, as long as the data is not available through the API, and as long as there is no such converter tool, users just have no option to get their annotations out of Mendeley in the standard PDF format.

    You are also incorrect on the following points: “Mendeley, for instance, will make it difficult (if
    not to say impossible) to collaborate with researchers using a reference manager other than
    Mendeley.” In fact, we have put extensive resources into developing CSL as an open standard,
    enabling anyone to add citation formatting to any application. See http://csl.mendeley.com/ Did
    you just miss this or did you mean something else?

    I am aware of Mendeley’s efforts to support CSL and I am very grateful for it. Mendeley’s Citation Style Editor is of great value for the entire academic community (and I know that Mendeley is doing much more for CSL). However, I wasn’t referring to CSL in my post. I mean the following: Let’s assume you are using JabRef which stores its bibliographic data in a BibTeX file. You want to collaborate with a colleague who is not using JabRef but another reference manager, e.g. BibDesk which, fortunately, also stores its data as BibTeX. Now, collaboration is really easy. You store your BibTeX file in DropBox and share it with your colleague. Any changes you make, your colleague will have within seconds, and vice versa. Because Mendeley is storing its data in a local database, in a proprietary format, you couldn’t do something similar with Mendeley. In addition, you couldn’t just store your PDFs in Dropbox and annotate them with Mendeley while your colleague is annotating them with a standard PDF editor (because Mendeley is not storing annotations in the standard PDF format). This means, if you are using Mendeley and want to work with a colleague on the same bibliographic data, or PDFs, both must use Mendeley’s premium services. They can’t use Dropbox to share their data (or another third party solution). As I wrote in my post: Mendeley’s premium services are great and allow a much more comprehensive collaboration than sharing a BibTeX file via Dropbox. Anyway, from a user perspective it is certainly far from optimal when you only have the choice between a) paying the price that Mendeley demands for collaboration (and convince your collaborators to do the same) or b) don’t collaborate. From a business perspective it’s great, though :-).

    “In addition, Mendeley can export bibliographic data to BibTeX but cannot import it.”
    Mendeley imports Bibtex quite well, actually.

    I am sorry. My statement was incorrect, indeed. What I wanted to say is that you cannot export a BibTex file, a colleague changes the BibTeX file and then you import the BibTeX file and the changes are reflected in Mendeley. Instead, changed documents are imported as new documents. For instance, if you have “Doc A” in Mendeley, you export it to BibTeX, a colleague changes the title to “Document A”, you import the BibTeX file, then you have “Doc A” and “Document A” in Mendeley. This problem relates to my criticism that with Mendeley you can only collaborate with other researchers who also use Mendeley. Anyway, I should have written this more clearly, I am sorry for the mistake, and will change my post.

    I always think twice or three times about it, and then get a neutral friend to read over it, just to
    make sure it doesn’t seem like I’m unfairly slamming the competition.

    I am really sorry, if my post seems like “unfair slamming” to you. This wasn’t my intention, but I still cannot see where I am treating Mendeley unfairly. To the best of my knowledge:

    A) Mendeley doesn’t export annotations in bulk, nor in the PDF standard format, but in some awkward proprietary format that does not allow to delete or edit annotations, or to see them where they were created but on the very last page. Hence, users cannot really use their annotations in other tools than Mendeley.

    B) You cannot share your bibliographic data and PDFs with researchers who are not using Mendeley, and work on this data collaboratively. This means, if you want to collaborate, you are forced to buy Mendeley’s premium services, and pay the price whatever it is (or you don’t collaborate).

    C) Mendeley does not export the virtual folder structure in which users organize the PDFs.

    Please let me know when you think that these points are not correct. However, if they are correct, then I cannot agree with your statement that “There is absolutely no lock-in with Mendeley whatsoever”. In my opinion, due to A) and C), Mendeley is locking its users in (in particular those who create annotations) and, due to B), Mendeley makes it difficult (if not to say impossible) to collaborate with researchers using a reference manager other than Mendeley.

Literaturverwaltung kompakt 1/2014 – Communitynachrichten | Literaturverwaltung · 21st February 2014 at 12:20

[…] Produktblog schon mehrfach die Frage nach der “besten” Software gestellt. (Vgl. etwa  hier, und hier) Mitte Januar erschien als jüngster Vergleichsbeitrag von Jöran Beel eine sehr […]

Literaturverwaltung kompakt 7/2013 – zweiter Teil: Communitynachrichten | Literaturverwaltung · 1st December 2013 at 17:48

[…] einen Hinweis auf seinen im Docear-Blog erschienenen und rege kommentierten Post “What makes a bad reference manager?“. In diesem stehen drei Hauptkriterien im […]

Wozu braucht man Docear? Argumente aus der Entwicklerperspektive | Literaturverwaltung · 16th November 2013 at 11:57

[…] Die wenigsten zweifeln dann noch, dass es einem Bedarf für eine weitere Literaturverwaltungssoftware gibt. Viele probieren Docear aus, viele sind begeistert, manche finden Docear zu komplex, manchen fehlen bestimmte Funktionen. Zotero Nutzer bemängeln beispielsweise oft Docears fehlenden Web-Importer. Mendeley Nutzer wünschen sich eine bessere Metadatenextraktion aus PDF Dateien. Eine solche gibt es zwar in Docear aber sie ist, zugegeben, nicht so komfortabel und gut wie die von Mendeley. Ein Add-On für LibreOffice ist derzeit auch noch nicht verfügbar (aber in Planung). Die Web-Version von Docear ist bisher sehr schlicht, und Mobile Apps gibt es gar nicht. Wer eine komfortable Lösung zur Kooperation möchte (Dateisynchronisierung, private Gruppen, …), und bereit ist dafür zu zahlen, wird bei Docear ebenfalls enttäuscht und sollte sich  besser die Angebote von Zotero, Mendeley, etc. anschauen. Dafür kann man Docears Daten mit Tools wie Dropbox synchronisieren, etwas das bei Mendeley und Zotero schwierig ist. […]

Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *