How we grade · Zotero Plugins Atlas

The ladder

A+ABC

A+Recommended, tested in ZoteroPassed the code checks, and behaved as its card describes installed in a live Zotero · 236 pluginsARecommendedEvery check of the code came out low · 298 pluginsBUse with careRead why before installing · 535 pluginsCNot recommendedWe found serious problems · 51 plugins–Not graded yetWe couldn't check it yet · 6 plugins

An A doesn't mean we found nothing. Small findings, such as storing an API key in Zotero's settings, are rated low and listed on the card; an A means none of them is more than that. A B means at least one finding is worth reading before you install. A C means we found something serious, such as code deliberately scrambled so nobody can check it.

What we check in the code

Code transparencyCan the code be read, and was the release file built from the public source?Sets the grade

Where your data goesWhich servers it contacts, and whether they're named services or the developer's own.Sets the grade

Powerful capabilitiesNative code, launching programs, running code built at runtime, reading files, storing passwords.Sets the grade

Works withWhich Zotero versions it installs on and runs in.Shown, not graded

What you'll needAPI keys, accounts or other software.Shown, not graded

MaintenanceHow recently it was released, and by how many people.Shown, not graded

LanguagesDocumentation and interface languages.Shown, not graded

Every finding links to the file and line it came from. The rules are published with the code, and they're the same for every plugin: when we change one, every card is recomputed.

The live test, for A+

Reading code shows what a plugin could do. The live test shows what it actually does.

1We install the exact release file in a fresh copy of Zotero 10, with a small sample library.

2We select items, open the reader and the plugin's settings, and click its menu items.

3We record every server it contacts and what it sends, including any of the sample library's text.

4If it loaded, and everything it did is something its card already describes, an A becomes A+.