Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

geopandas 1.0.0 or pyogrio 0.9.0 doesn't support OGR_GMT output? #3301

Open
seisman opened this issue Jun 25, 2024 · 7 comments
Open

geopandas 1.0.0 or pyogrio 0.9.0 doesn't support OGR_GMT output? #3301

seisman opened this issue Jun 25, 2024 · 7 comments
Labels
bug Something isn't working upstream Bug or missing feature of upstream core GMT

Comments

@seisman
Copy link
Member

seisman commented Jun 25, 2024 •
edited
Loading

geopandas v1.0.0 was released a few hours ago and now our Docs workflow fails to build (https://github.com/GenericMappingTools/pygmt/actions/runs/9559533732/job/26634813440) which is not caught in our "GMT Dev Tests" workflow (why?):

Extension error:
Here is a summary of the problems encountered when running the examples:

Unexpected failing examples:

    ../examples/get_started/04_table_inputs.py failed leaving traceback:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/get_started/04_table_inputs.py", line 111, in <module>
        fig.plot(data=gdf, style="c0.2c", fill="purple")
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/decorators.py", line 609, in new_module
        return module_func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/decorators.py", line 773, in new_module
        return module_func(*bound.args, **bound.kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/src/plot.py", line 258, in plot
        with lib.virtualfile_in(
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/contextlib.py", line 137, in __enter__
        return next(self.gen)
               ^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/tempfile.py", line 164, in tempfile_from_geojson
        geojson.to_file(**ogrgmt_kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/geodataframe.py", line 1536, in to_file
        _to_file(self, filename, driver, schema, index, **kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 686, in _to_file
        _to_file_pyogrio(df, filename, driver, schema, crs, mode, metadata, **kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 748, in _to_file_pyogrio
        pyogrio.write_dataframe(df, filename, driver=driver, metadata=metadata, **kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/geopandas.py", line 456, in write_dataframe
        path, driver = _get_write_path_driver(path, driver, append=append)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/raw.py", line 564, in _get_write_path_driver
        if not ogr_driver_supports_write(driver):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "pyogrio/_ogr.pyx", line 105, in pyogrio._ogr.ogr_driver_supports_write
      File "pyogrio/_ogr.pyx", line 287, in pyogrio._ogr._get_driver_metadata_item
    pyogrio.errors.DataSourceError: Could not obtain driver: OGR_GMT (check that it was installed correctly into GDAL)

    ../examples/gallery/lines/roads.py failed leaving traceback:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/gallery/lines/roads.py", line 19, in <module>
        gdf = gpd.read_file(
              ^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 294, in _read_file
        return _read_file_pyogrio(
               ^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 547, in _read_file_pyogrio
        return pyogrio.read_dataframe(path_or_bytes, bbox=bbox, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/geopandas.py", line 261, in read_dataframe
        result = read_func(
                 ^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/raw.py", line 196, in read
        return ogr_read(
               ^^^^^^^^^
      File "pyogrio/_io.pyx", line 1239, in pyogrio._io.ogr_read
      File "pyogrio/_io.pyx", line 219, in pyogrio._io.ogr_open
    pyogrio.errors.DataSourceError: No driver registered.
@seisman seisman added the bug Something isn't working label Jun 25, 2024
@seisman
Copy link
Member Author

seisman commented Jun 26, 2024 •
edited
Loading

That's weird. I tried to create the same environment locally.

mamba create --name pygmt-docs python=3.12 gmt=6.5.0 ghostscript=10.03.1 numpy pandas xarray netCDF4 packaging contextily geopandas ipython rioxarray build make pip myst-parser panel sphinx sphinx-autodoc-typehints sphinx-copybutton sphinx-design sphinx-gallery sphinx_rtd_theme

Now I have geopandas 1.0.0, pyogrio 0.9.0 and gdal 3.9.0 installed, but the script works well for me.

In [1]: import pygmt

In [2]: import geopandas as gpd

In [3]: # Example GeoDataFrame
   ...: gdf = gpd.GeoDataFrame(
   ...:     {
   ...:         "geometry": gpd.points_from_xy([2, 5, 9], [2, 3, 4]),
   ...:         "value": [10, 20, 30],
   ...:     }
   ...: )
   ...:
   ...: # Use the GeoDataFrame to specify the 'data' parameter
   ...: fig = pygmt.Figure()
   ...: fig.basemap(region=[0, 10, 0, 5], projection="X10c/5c", frame=True)
   ...: fig.plot(data=gdf, style="c0.2c", fill="purple")
   ...: fig.show()

@seisman
Copy link
Member Author

seisman commented Jun 26, 2024

There is one more failure:

../examples/gallery/lines/roads.py unexpectedly failed to execute correctly:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/gallery/lines/roads.py", line 19, in <module>
        gdf = gpd.read_file(
              ^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 294, in _read_file
        return _read_file_pyogrio(
               ^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 547, in _read_file_pyogrio
        return pyogrio.read_dataframe(path_or_bytes, bbox=bbox, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/geopandas.py", line 261, in read_dataframe
        result = read_func(
                 ^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/raw.py", line 196, in read
        return ogr_read(
               ^^^^^^^^^
      File "pyogrio/_io.pyx", line 1239, in pyogrio._io.ogr_read
      File "pyogrio/_io.pyx", line 219, in pyogrio._io.ogr_open
    pyogrio.errors.DataSourceError: No driver registered.

@seisman
Copy link
Member Author

seisman commented Jun 26, 2024

pyogrio.errors.DataSourceError: No driver registered.

This error comes from GDAL https://github.com/OSGeo/gdal/blob/a66c71d0e1e19a5ef9d94b33bef90074173be9c2/gcore/gdaldataset.cpp#L3965C7-L3965C7

This was referenced Jun 27, 2024
Merged
Draft
@weiji14
Copy link
Member

weiji14 commented Jul 9, 2024

geopandas v1.0.0 was released a few hours ago and now our Docs workflow fails to build (https://github.com/GenericMappingTools/pygmt/actions/runs/9559533732/job/26634813440) which is not caught in our "GMT Dev Tests" workflow (why?)

The "GMT Dev Tests" installs geopandas from PyPI, whereas the Docs workflow installs geopandas from conda-forge. Are there differences in GDAL version used? I'm reading https://github.com/geopandas/pyogrio/blob/v0.9.0/docs/source/install.md#troubleshooting-installation-errors, and it says errors like pyogrio.errors.DataSourceError: Could not obtain driver ... may be due to incompatible versions of GDAL in fiona and pyogrio, but we don't have fiona installed anymore 😕

Your example at #3301 (comment) works for me too if I run it individually. But when I do cd docs; make clean; make all to run the sphinx build, it fails the first time, but works if I run it twice. Looks like there's some flaky behaviour going on.

@seisman
Copy link
Member Author

seisman commented Jul 9, 2024

The "GMT Dev Tests" installs geopandas from PyPI, whereas the Docs workflow installs geopandas from conda-forge. Are there differences in GDAL version used?

The "Tests" workflow also has geopandas from conda-forge, but the geopandas tests all pass.

@weiji14
Copy link
Member

weiji14 commented Jul 9, 2024 •
edited
Loading

Found a related issue upstream at geopandas/pyogrio#144. There is supposed to be some logic to register GDAL drivers when pyogrio is imported (geopandas/pyogrio#145), but it is not working properly in Sphinx build for some reason, and causing that pyogrio.errors.DataSourceError: Could not obtain driver: OGR_GMT (check that it was installed correctly into GDAL) error. Note that the OGR_GMT driver does exist in our GDAL installation, just that the driver isn't loaded properly.

Workaround is to manually register the drivers like so:

import pyogrio

pyogrio.core._register_drivers()

See also https://github.com/GenericMappingTools/pygmt/pull/3305/files#r1669745089 where this patch is applied.

@weiji14 weiji14 mentioned this issue Jul 9, 2024
Open
@weiji14
Copy link
Member

weiji14 commented Jul 10, 2024 •
edited
Loading

Have reported this upstream, and got a super helpful response at geopandas/pyogrio#448 (comment). It seems like there may be some conflict with the order of how rioxarray/rasterio and pyogrio is loading the GDAL drivers.

@weiji14 weiji14 added the upstream Bug or missing feature of upstream core GMT label Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working upstream Bug or missing feature of upstream core GMT
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@seisman @weiji14