Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Geonode 4.0.2 with geonode-importer does not upload #12618

Open
fabiomonticelli23 opened this issue Sep 27, 2024 · 3 comments
Open

Geonode 4.0.2 with geonode-importer does not upload #12618

fabiomonticelli23 opened this issue Sep 27, 2024 · 3 comments

Comments

@fabiomonticelli23
Copy link

fabiomonticelli23 commented Sep 27, 2024

Expected Behavior

After changing the settings.py with the geonode.importer celery tasks it should use the new API and upload but it returns an error

Actual Behavior

After calling the API and passing a simple tiff file it return this error
image

Steps to Reproduce the Problem

1.installed Geonode 4.0.2 using docker compose
2. added in dockerfile after pip install requirements -> RUN pip install -e git+https://github.com/geosolutions-it/[email protected]#egg=geonode_importer
3.Added at the end of settings.py

INSTALLED_APPS += ('dynamic_models', 'importer', 'importer.handlers',)

CELERY_TASK_QUEUES += (
    Queue('importer.import_orchestrator', GEONODE_EXCHANGE, routing_key='importer.import_orchestrator'),
    Queue('importer.import_resource', GEONODE_EXCHANGE, routing_key='importer.import_resource', max_priority=8),
    Queue('importer.publish_resource', GEONODE_EXCHANGE, routing_key='importer.publish_resource', max_priority=8),
    Queue('importer.create_geonode_resource', GEONODE_EXCHANGE, routing_key='importer.create_geonode_resource', max_priority=8),
    Queue('importer.import_with_ogr2ogr', GEONODE_EXCHANGE, routing_key='importer.import_with_ogr2ogr', max_priority=10),
    Queue('importer.import_next_step', GEONODE_EXCHANGE, routing_key='importer.import_next_step', max_priority=3),
    Queue('importer.create_dynamic_structure', GEONODE_EXCHANGE, routing_key='importer.create_dynamic_structure', max_priority=10),
    Queue('importer.copy_geonode_resource', GEONODE_EXCHANGE, routing_key='importer.copy_geonode_resource', max_priority=0),
    Queue('importer.copy_dynamic_model', GEONODE_EXCHANGE, routing_key='importer.copy_dynamic_model'),
    Queue('importer.copy_geonode_data_table', GEONODE_EXCHANGE, routing_key='importer.copy_geonode_data_table'),
    Queue('importer.copy_raster_file', GEONODE_EXCHANGE, routing_key='importer.copy_raster_file'),
    Queue('importer.rollback', GEONODE_EXCHANGE, routing_key='importer.rollback'),
)

DATABASE_ROUTERS = ["importer.db_router.DatastoreRouter"]

SIZE_RESTRICTED_FILE_UPLOAD_ELEGIBLE_URL_NAMES += ('importer_upload',)

IMPORTER_HANDLERS = os.getenv('IMPORTER_HANDLERS', [
    'importer.handlers.gpkg.handler.GPKGFileHandler',
    'importer.handlers.geojson.handler.GeoJsonFileHandler',
    'importer.handlers.shapefile.handler.ShapeFileHandler',
    'importer.handlers.kml.handler.KMLFileHandler',
    'importer.handlers.csv.handler.CSVFileHandler',
    'importer.handlers.geotiff.handler.GeoTiffFileHandler'
])
FILE_UPLOAD_DIRECTORY_PERMISSIONS = 0o777
FILE_UPLOAD_PERMISSIONS = 0o777

Specifications

  • GeoNode version: 4.0.2
  • Installation type (vanilla, geonode-project):
  • Installation method (manual, docker): docker
  • Platform:
  • Additional details:
@fabiomonticelli23 fabiomonticelli23 changed the title Geonode 4.0.2 with geonode-impoter does not upload Geonode 4.0.2 with geonode-importer does not upload Sep 27, 2024
@mattiagiupponi
Copy link
Contributor

mattiagiupponi commented Sep 30, 2024

geonode-importer was introduced for GeoNode>=4.1.x if i'm not wrong ( #10474 ) so is not guarantee that works with Geonode 4.0.2

Aside this, the importer logs more info in the log when something happen. Please try to re-upload the tif and attach the celery.log file from the celery container. The log is located under /var/log/celery.log

Plus the dockerfile between the introduction of the two versions has changed as you can see from this:
GeoNode/geonode-project@7a3d31f#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R5

So i suppose that gdal-bin is missing in your dockerfile as dependency

@fabiomonticelli23
Copy link
Author

fabiomonticelli23 commented Sep 30, 2024

this is the content of celery.log as you can see it doesnt find the db['ENGINE'] value

[2024-09-30 08:34:02,080: ERROR/ForkPoolWorker-10] Task importer.publish_resource[00e97085-7ed9-4a69-a0fb-b1ada77e3add] raised unexpected: PublishResourceException()
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/geonode/geoserver/helpers.py", line 1421, in create_geoserver_db_featurestore
    raise FailedRequestError
geoserver.catalog.FailedRequestError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/importer/celery_tasks.py", line 236, in publish_resource
    _publisher.publish_resources(data)
  File "/usr/local/lib/python3.10/site-packages/importer/publisher.py", line 63, in publish_resources
    self.get_or_create_store()
  File "/usr/local/lib/python3.10/site-packages/importer/publisher.py", line 93, in get_or_create_store
    self.store = create_geoserver_db_featurestore(
  File "/usr/local/lib/python3.10/site-packages/geonode/geoserver/helpers.py", line 1427, in create_geoserver_db_featurestore
    db_engine = "postgis" if "postgis" in db["ENGINE"] else db["ENGINE"]
KeyError: 'ENGINE'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/importer/celery_tasks.py", line 282, in publish_resource
    raise PublishResourceException(detail=error_handler(e, execution_id))
importer.api.exception.PublishResourceException: ENGINE. Request: 44cb6eae-51a6-4396-b051-79744bd5a3b6
[2024-09-30 08:34:02,080: INFO/MainProcess] Task importer.rollback[c792a367-e0fc-4716-a292-a0cb3c6b6ea0] received
[2024-09-30 08:34:02,081: DEBUG/MainProcess] TaskPool: Apply <function fast_trace_task at 0x7f6913d2b640> (args:('importer.rollback', 'c792a367-e0fc-4716-a292-a0cb3c6b6ea0', {'lang': 'py', 'task': 'importer.rollback', 'id': 'c792a367-e0fc-4716-a292-a0cb3c6b6ea0', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '7f2484a1-113f-4104-88ef-6de239a09500', 'parent_id': 'a5b2cd4a-9547-40bd-90f4-06b8eb546cd6', 'argsrepr': "('44cb6eae-51a6-4396-b051-79744bd5a3b6', 'importer.rollback', 'test_local_machine_5_aoi', 'test_local_machine_5_aoi', 'importer.handlers.geojson.handler.GeoJsonFileHandler', 'rollback')", 'kwargsrepr': "{'kwargs': {'kwargs': {}, 'previous_action': 'import', 'error': 'ENGINE. Request: 44cb6eae-51a6-4396-b051-79744bd5a3b6'}}", 'origin': 'gen178@ff61406f96fd', 'ignore_result': True, 'properties': {'content_type': 'application/json', 'content_encoding': 'utf-8', 'application_headers': {'lang': 'py', 'task': 'importer.rollback', 'id': 'c792a367-e0fc-4716-a292-a0cb3c6b6ea0', 'shadow': None, 'eta': None, 'expires': None, 'group':... kwargs:{})
[2024-09-30 08:34:02,082: INFO/ForkPoolWorker-10] Calling rollback for execution_id 44cb6eae-51a6-4396-b051-79744bd5a3b6 in progress
[2024-09-30 08:34:02,088: INFO/ForkPoolWorker-12] Task importer.import_orchestrator[a5b2cd4a-9547-40bd-90f4-06b8eb546cd6] succeeded in 0.054907314000047336s: None
[2024-09-30 08:34:02,097: WARNING/ForkPoolWorker-10] Starting rollback for execid: 44cb6eae-51a6-4396-b051-79744bd5a3b6 resource published was: test_local_machine_5_aoi
[2024-09-30 08:34:02,097: INFO/ForkPoolWorker-10] Rollback publishing step in progress for execid: 44cb6eae-51a6-4396-b051-79744bd5a3b6 resource published was: test_local_machine_5_aoi
[2024-09-30 08:34:02,145: INFO/ForkPoolWorker-10] Rollback dynamic model & ogr2ogr step in progress for execid: 44cb6eae-51a6-4396-b051-79744bd5a3b6 resource published was: test_local_machine_5_aoi
[2024-09-30 08:34:02,145: INFO/ForkPoolWorker-10] Dynamic model does not exists, removing ogr2ogr table in progress
[2024-09-30 08:34:02,166: INFO/ForkPoolWorker-10] table "test_local_machine_5_aoi" does not exist

for gdal-bin i modified the dockerfile to install it

@mattiagiupponi
Copy link
Contributor

it looks like an issue with the generation of the geonode_data store in geoserver, can you try to create it manually?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants