neonutilities submission

[cklunch]

Submitting Author: Claire Lunch (@cklunch)
All current maintainers: (@cklunch, @bhass-neon, @znickerson8)
Package Name: neonutilities
One-Line Description of Package: neonutilities is a package for accessing and wrangling data generated and published by the National Ecological Observatory Network.
Repository Link: https://github.com/NEONScience/NEON-utilities-python
Version submitted: v1.0.1
EiC: @cmarmo
Editor: TBD
Reviewer 1: TBD
Reviewer 2: TBD
Archive: TBD
JOSS DOI: TBD
Version accepted: TBD
Date accepted (month/day/year): TBD

---

Code of Conduct & Commitment to Maintain Package

• I agree to abide by pyOpenSci's Code of Conduct during the review process and in maintaining my package after should it be accepted.
• I have read and will commit to package maintenance after the review as per the pyOpenSci Policies Guidelines.

Description

The neonutilities Python package provides utilities for discovering, downloading, and working with data files published by the National Ecological Observatory Network (NEON). NEON data files can be downloaded from the NEON Data Portal or API. The neonutilities package includes wrapper functions for the API and functions to reformat and stack NEON tabular data for analysis. This is a Python-native adaptation of the heavily used neonUtilities R package.

Scope

• Please indicate which category or categories.
  Check out our package scope page to learn more about our
  scope. (If you are unsure of which category you fit, we suggest you make a pre-submission inquiry):

  • Data retrieval
  • Data extraction
  • Data processing/munging
  • Data deposition
  • Data validation and testing
  • Data visualization¹
  • Workflow automation
  • Citation management and bibliometrics
  • Scientific software wrappers
  • Database interoperability

Domain Specific

• Geospatial
• Education

Community Partnerships

If your package is associated with an
existing community please check below:

• Astropy:My package adheres to Astropy community standards
• Pangeo: My package adheres to the Pangeo standards listed in the pyOpenSci peer review guidebook

• For all submissions, explain how and why the package falls under the categories you indicated above. In your explanation, please address the following points (briefly, 1-2 sentences for each):

  • Who is the target audience and what are scientific applications of this package?

The target audience is scientists doing research using NEON data. The package enables programmatic workflows for data downloading, and provides a standardized way to merge the product-site-month data files NEON publishes, making them analysis-ready.

• Are there other Python packages that accomplish the same thing? If so, how does yours differ?

There is an incomplete package on PyPi here that was started in 2020 by a student at a coding camp. It doesn't appear to have been finished, and is not maintained. Some NEON users have developed their own code to do some of the functionality covered by neonutilities, but as far as I know none of them have shared it broadly.

• If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted:

Technical checks

For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box. This package:

• does not violate the Terms of Service of any service it interacts with.
• uses an OSI approved license.
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions.
• contains a tutorial with examples of its essential functions and uses.
• has a test suite.
• has continuous integration setup, such as GitHub Actions CircleCI, and/or others.

Publication Options

• Do you wish to automatically submit to the Journal of Open Source Software? If so:

JOSS Checks

• The package has an obvious research application according to JOSS's definition in their submission requirements. Be aware that completing the pyOpenSci review process does not guarantee acceptance to JOSS. Be sure to read their submission requirements (linked above) if you are interested in submitting to JOSS.
• The package is not a "minor utility" as defined by JOSS's submission requirements: "Minor ‘utility’ packages, including ‘thin’ API clients, are not acceptable." pyOpenSci welcomes these packages under "Data Retrieval", but JOSS has slightly different criteria.
• The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
• The package is deposited in a long-term repository with the DOI:

Note: JOSS accepts our review as theirs. You will NOT need to go through another full review. JOSS will only review your paper.md file. Be sure to link to this pyOpenSci issue when a JOSS issue is opened for your package. Also be sure to tell the JOSS editor that this is a pyOpenSci reviewed package once you reach this step.

Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?

This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.

• Yes I am OK with reviewers submitting requested changes as issues to my repo. Reviewers will then link to the issues in their submitted review.

Confirm each of the following by checking the box.

• I have read the author guide.
• I expect to maintain this package for at least 2 years and can help find a replacement for the maintainer (team) if needed.

Please fill out our survey

• Last but not least please fill out our pre-review survey. This helps us track
  submission and improve our peer review process. We will also ask our reviewers
  and editors to fill this out.

P.S. Have feedback/comments about our review process? Leave a comment here

Editor and Review Templates

The editor template can be found here.

The review template can be found here.

Footnotes

1. Please fill out a pre-submission inquiry before submitting a data visualization package. ↩

[cmarmo]

Editor in Chief checks

Hi @cklunch! Thank you for submitting your package for pyOpenSci review.
Below are the basic checks that your package needs to pass to begin our review.
If some of these are missing, we will ask you to work on them before the review process begins.

Please check our Python packaging guide for more information on the elements below.

• Installation The package can be installed from a community repository such as PyPI (preferred), and/or a community channel on conda (e.g. conda-forge, bioconda).
  • The package imports properly into a standard Python environment import package.
• Fit The package meets criteria for fit and overlap.
• Documentation The package has sufficient online documentation to allow us to evaluate package function and scope without installing the package. This includes:
  • User-facing documentation that overviews how to install and start using the package.
  • Short tutorials that help a user understand how to use the package and what it can do for them.
  • API documentation (documentation for your code's functions, classes, methods and attributes): this includes clearly written docstrings with variables defined using a standard docstring format.
• Core GitHub repository Files
  • README The package has a README.md file with clear explanation of what the package does, instructions on how to install it, and a link to development instructions.
  • Contributing File The package has a CONTRIBUTING.md file that details how to install and contribute to the package.
  • Code of Conduct The package has a CODE_OF_CONDUCT.md file.
  • License The package has an OSI approved license.
    NOTE: We prefer that you have development instructions in your documentation too.
• Issue Submission Documentation All of the information is filled out in the YAML header of the issue (located at the top of the issue template).
• Automated tests Package has a testing suite and is tested via a Continuous Integration service.
• Repository The repository link resolves correctly.
• Package overlap The package doesn't entirely overlap with the functionality of other packages that have already been submitted to pyOpenSci.
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly.
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

• Initial onboarding survey was filled out
  We appreciate each maintainer of the package filling out this survey individually. 🙌
  Thank you authors in advance for setting aside five to ten minutes to do this. It truly helps our organization. 🙌

---

---

Editor comments

Here my comments to clarify why I haven't ticked all the check-boxes.

• Documentation: I've found a lot of relevant information in the https://www.neonscience.org/ page, however there is no standalone documentation for the package, even if a docs directory is present in the repo with an empty index.md. I strongly recommend to add some documentation pages in order to structure the access to the information. There is no need to rewrite what is already on https://www.neonscience.org/, just linking and structuring, following for example the structure suggested in the pre-review checks. Also, it is not clear to me what is part of the Neon Data API and what is part of the python implementation.
• The README does not contain the documentation about how to contribute: it is true that the CONTRIBUTING file specifies that external contributions will not be considered for now. Perhaps a link to the CONTRIBUTING would be enough.
• As opening issues is the only way to contribute for now, it would be nice to have issue templates for different kind of contributions (bug reports, new feature suggestions, documentation,...)

I think the fluidity of the review process will benefit from those improvements.

Thank you so much for your understanding!

[lwasser]

hey there @cklunch @bhass-neon @znickerson8 👋🏻 it's nice to see you here on GitHub! I have a question for you about your contributing doc which says:

The neonutilities package is currently not accepting external contributions. If you have a suggestion for a fix or enhancement to the package, please create an Issue in this repository, or contact us using the NEON Contact Us page.

We do consider opening issues to report bugs and to request useful features to be contributions. They are a different type than a PR of course. What is the intent of "we are not accepting contributions" is in this case? We are having some discussions around our pyOpenSci policies, maintainer responsiveness to users and open source processes in general related to this package. Any input from you would be super helpful! thank you!!

[cklunch]

@cmarmo @lwasser Thanks for the feedback!

For documentation, the intent was for the tutorials that are linked in the readme to provide the documentation users need to understand how to use the package. The Download and Explore tutorial provides instructions in the most commonly used functions, and some context and common follow-up data wrangling, and the neonUtilities tutorial provides a function index. If this isn't what you're looking for, or if different content is needed, do you have an example of documentation that would be appropriate?

For contributing, of course bug reports and requests are always welcome! We've found with the R package that it isn't realistic for external folks to contribute code directly. The package is so specific to the NEON publication system, it's hard for people who aren't deeply familiar with that system to write generic code for it. It's almost always easier for us to incorporate requested changes ourselves than to work with external PRs. But if there's a better way to express that in the documentation, I'm very happy to.

I'll get issue templates set up in the repo, and wait to update documentation based on what we decide in this discussion. Thanks!

[lwasser]

hey there @cklunch

Have a look at our packaging guide here. Typically, python packages have a documentation "website". You might also have a look at some of the structures for other packages in our ecosystem.

tutorials are great, but it's also important to document the code base for easier future maintenance, help people get started with installing the package, etc. Please have a look at those resources and let us know if you have any questions!

[cmarmo]

For documentation, the intent was for the tutorials that are linked in the readme to provide the documentation users need to understand how to use the package. The Download and Explore tutorial provides instructions in the most commonly used functions, and some context and common follow-up data wrangling, and the neonUtilities tutorial provides a function index. If this isn't what you're looking for, or if different content is needed, do you have an example of documentation that would be appropriate?

Indeed, that's why I said that a lot of information is already there. The way it is structured though is not easy to browse for someone new to the neon ecosystem.
It would be easier also for the review, to have a reference documentation page (like your index.md), rendered by github pages or similar, with a table of contents and the essential information: Getting started, tutorials, API, and then the links to the related pages already available in the neon website.
Does that sound reasonable?

[cklunch]

@lwasser @cmarmo Thanks for the clarification. I'll see about creating a dedicated page.