Athena to iceberg method not writting data to columns that are new in the schema

[lautarortega]

Describe the bug

I have a table that was created by a glue job. I want to append data to that table using AWS Wrangler. The writting process seems to work fine, but when I check on Athena, the columns that were not there before are added but appear to be completely empty, while there were no nulls in my dataframe.

If I delete the rows I appended and write the data again using AWS Wrangler, the table is updated correctly, since the columns are not new anymore.

How to Reproduce

I tried replicating the issue using just AWS Wrangler and I could not do it.
Try having a glue job create an iceberg table and then try to update this table with an extra column using wrangler.

Expected behavior

No response

Your project

No response

Screenshots

No response

OS

Mac

Python version

3.10

AWS SDK for pandas version

3.9.0

Additional context

No response