Merge multiple databases at once with the option to have redundant databases deleted.

[rtevans01]

I use Syncthing to sync my database with the all devices I use. Unfortunately, I often run into the issue of getting conflicting databases. I think this is solely due to the nature of Syncthing since it's not centralized and can get messy.

I'm requesting the existing merging feature in KeePassXC be expanded to open multiple conflicting databases at once, merge them, and then automatically delete them afterwards. This would make maintenance easier for me. I don't know if this is a common issue for most other users though.

EDIT: I just had an idea. How about allowing an option to open the regular database as well as all the conflicting databases from the unlocked screen? That would save a step.

[aisbergde]

@rtevans01
How keepassxc should know which databases belongs together?

My main issues also are related to synchronizing databases on different devices using cloud services.

Maybe the request could be more general:

• define databases which belongs together (maybe using some syntax in filenames)
• open all databases which belongs together in different tabs, requesting the password only one time
• allow merging between all opened databases, not requesting a password again
• add a context menu to tabs of opened database:
  • close tab
  • delete database
  • ...

[rtevans01]

Yes I think a syntax should be used. My conflicted databases are all in the same directory path as my non-conflicted database, so I don't think it would be very difficult to search for them. Just start out with the name of the non-conflicted database and look for any other database that has the same name plus "sync-conflict-2020...".

I think opening separate tabs would be a waste of time. I'd rather have it taken care of in one step but to each his own.

[boernsen-development]

After reading a lot of related issues, I think there is quiet some confusion in the acual issues and responsibilities related to cloud syncing in combination with KeePassXC, but I think this issue hits it best, so I am adding some details to it. After analysing it for quiet some time now, let me use this comment to document it in detail and hopefully also motivate for a new feature of KeePassXC as I think it will affect many users out there.

I think using KeePassXC on multiple devices accessing a single remote kdbx file works seamlessly as long as direct file system access to the actual and only kdbx file is used (such as an SMB share)! Changes done to the file while still open in KeePassXC can be automatically merged if configured. If the file is not accessible anymore on the other hand, KeePassXC will notify you. It therefore only works properly though in local networks or anywhere where connectivity to the original kdbx file is stable, so that you can use a file system based access to the file. We use this method in a team already since a year and I recommend everyone who is able to use this method to use it as it's easy to setup, very reliable and has great usability as changes are merged immediately. Thank you for this awesome feature!

However, at the point where KeePassXC is supposed to be used with cloud sync (like Dropbox, Syncthing, etc.), there is one major issue making it very annoying to use as described already by @rtevans01. It is actually not a bug, but a major usability issue due to the nature of how cloud sync works and how tasks/responsibilities are distributed across different software (such as KeePass and a sync software in this case).

The main issue are conflicts happening whenever two devices do not have a continuous connection. See this example with devices A and B:

• A and B both have a connection, their sync'ed kdbx files are up-to-date
• A and B both loose connection
• Both make any update or change and save their file locally (independent which entry they change or whether they add or remove something)
• Now the sync provider tries to sync the files on both devices
• Let's say A comes first for whatever reason and succeeds syncing
• B tries the same, but cannot sync to the actual kdbx file anymore since it has changed already. This is called a sync conflict, so the sync tool usually prevents data loss by creating a conflict file next to the actual file containing the local changes, which then is synced to both devices, A and B again. The actual kdbx (with A's changes) is then synced to B also causing B's KeePassXC to detect the file changes (if still opened) and asking for a reload or even reloading it automatically if configured. However, the actual kdbx now only contains A's changes and from B's point of view B's local changes seem to have disappeared suddenly. [#usability_issue1] They are still in the conflict file of course.
• Now, A and B have two versions of the kdbx, an original (containing the changes from A) and a conflict file (containing the changes from B)
• To resolve this conflict, firstly, either A or B needs to actually notice the existence of a conflict file [#usability_issue2] and then manually merge it into the actual kdbx, save the actual kdbx and delete the conflict kdbx. [#usability_issue3]
• The merged kdbx will then be synced to the other device again and both are in sync again.

The general situation is not unusual as in practice the connection between the device and a cloud server is not always reliable. Think of mobile phones or home-driven cloud servers for example. With a LAN-only Syncthing setup, this even is every day practice.

Note that I tagged some positions with #usability_issueX. I think the major usability issues are:

• [#usability_issue1] The automatic reload is correct if configured, but very confusing as the local changes just disappear. Not having the automatic reload makes distributed changes very inconvenient. If KeePassXC was closed during the sync conflict, B probably wouldn't even notice the revert except B manually checks for conflict files every time opening the database.
• [#usability_issue2] The conflict itself can be overseen easily since you have to manually check for conflict files
• [#usability_issue3] The manual merge/delete is inconvenient if you need to do it often and error prone if there is more than a single conflict file as you need to manually compare file names.

Note that the file name patterns for conflict files are well documented for the popular sync providers:

• https://docs.syncthing.net/users/syncing.html?highlight=conflict#conflict-handling
• <filename>.sync-conflict-<date>-<time>-<modifiedBy>.<ext>
• https://help.dropbox.com/files-folders/share/conflicted-copy
• “Conflicted copy”, the editor’s username, and the save date will be added to the file name.
• https://docs.nextcloud.com/desktop/2.5/conflicts.html or https://doc.owncloud.org/desktop/2.5/conflicts.html
• mydata (conflicted copy 2018-04-10 093612).txt containing “local contents”

I therefore would like to encourage to add new features to KeePassXC that increase the usability for this very regular usecase of combining KeePassXC with cloud sync. I am not too familiar with best practices and no-gos in terms of security, but here are some ideas I have:

• automatic detection of conflict files (eventually by a manually configurable file name pattern to put this responsibility to the user)
• automatic merge/move-to-bin of conflict files
• alternatively (if the actions are not supposed to be done automatically for whatever reason) at least automatic notifications/dialogs about the existence of conflict files, the merge and/or the move-to-bin action
• at very least a good documentation about the situation with cloud sync

I hope this clarifies and motivates for some solution of the usability issue.

That was a long run. Thanks for reading down to here! :)

I was tempted to open a new issue for that, but I think there is quiet some confusion already due to many related tasks and some of the aspects described are already somewhere else, so I don't wanna mess it up more. These are some related tasks which might be worth considering: #90, #2937, #4192, #5072, #5024.
What would be the best practice with this @droidmonkey?

[droidmonkey]

Can you TLDR the above

[aisbergde]

@boernsen1
You described this very clear. A "TLDR" makes not so much sense, because then some important content and context will be missing (IMHO). Thank you for this comment.

And that's why my solution with Keepass was (and is)

• to use a "master" file which is synced between devices.
• I don't change the master directly, but I change "working copies" on different devices.
• I explicitly "merge" (in keepass words: synchronize) my working copy with the master when the connection is good and there is a good change to get the master updated in the cloud.

That's why I still hope to get bi-directional merge.

And, to go back to my comment above: KeepassX should try to use the same password as for the current opened file. This is how Keepass works: I will not be asked to enter the password again for each file I want to synchronize. I can just "synchronize" and that's it.

[boernsen-development]

I'll try to sum up my indeed very long comment.

Current situation

• Sharing a kdbx with direct file access / single file (like SMB) works great and very convenient due to the automatic and robust merge feature!
• Sharing a kdbx with any cloud sync (Syncthing, Dropbox, Nextcloud, etc) in theory works, but is very inconvenient and prone to oversee things (thus almost unusable) due to regularly arising sync conflict files, created by the syncing software, which have to be merged manually in an inconvenient process in KeePassXC. For details why those conflict files arise so regularly, see my detailed example above.

Inconvenient due to

• manually and regularly checking for conflict files / sudden disappearing of changes just made because they are moved into a conflict file by the sync software
• manually selecting conflict files for merge
• manually entering the (identical) password of the conflict file
• manually deleting already merged conflict files by comparing complicated file names

Desired

• convenient handling (i.e. merging, deleting) of these conflict files (also see my long post for sync-software-specific patterns), preferably automatic, but at least semi-automatic using dialogs and notifications

[droidmonkey]

Great thank you

[boernsen-development]

Hey @droidmonkey, is there any news about that?

At least there should be a comment about it at: https://keepassxc.org/docs/#faq-cloudsync
It still suggests, there is nothing unusual to expect, but if you're not a technician, I claim one would not know where to find missing (conflicting) passwords.

Just to make it clear: cloud synchronisation currently only works flawlessly if only one device is writing and all others are reading only. Otherwise conflicts are very likely, especially with deferred sync tools like Syncthing.

[droidmonkey]

No there hasn't been any effort put forth on this one. Personally, I use cloud sync all the time and write to my database on mobile device and laptop, never had a conflict yet. This doesn't appear to be a common problem for our user base, this is the only discussion about sync conflicts we ever had.

[ekasprzak]

Can't agree with that - whenever I try (every few months in last few years) to switch back from Bitwarden (because of no auto-type function there) I ALWAYS get conflicts in matter of max few days, where most of the time it happens much quicker.
Fully agree with @boernsen1 that suggesting that it works great is just wrong.
Quick search on google (and even ddg) for keepassxc conflicts on sync brings thousands of results with a lot of people having those problems.
So, I disagree completely with "This doesn't appear to be a common problem for our user base"
It's a big blocker for me for switch back to Keepassxc (I did the switch few years ago exactly because of constant sync issues even when I'm trying hard to follow general advice - only one writing to db device at the time)

[droidmonkey]

You might be confusing the request to have a direct cloud sync solution with this request for conflict resolution. Also the original keepass does not have this conflict resolution either so it's not like we are alone here.

[aisbergde]

bad synchronization support is the main issue in KeepassXC.

The normal Keepass doesn't have this issue, because it can sync in both directions. And it doesn't require to reenter the password again and again, when both databases have the same password.

Keepass for Android also can sync in in both directions, but it needs to be online, and this is sometimes not the case.

Currently, you must be careful if you want to change the database on different devices and there is no connection to the cloud in the meantime. In fact, it only works if you make sure that changes to the database end up in the cloud immediately.

As soon as you change the database on any device that is not online, there are often problems. And it's when you're on the road and you change the database on a mobile device and the changes don't immediately end up in the cloud that the problems start.

Therefore, one solution would be to use a master database that is not changed directly, but with which device-specific databases are synchronized. For me would be OK, if this is not done in one step, but in 2 steps:

• merge from the other database
• merge into the other database

But then at least it should be recognized that both databases have the same password, so you must enter it only once: in the database you are working with.

[tomwj]

I've had this issue intermittently for the last couple of years using Syncthing.

This thread has been useful for some workarounds e.g. making sure at least one device is always on and running with Syncthing to avoid deferred sync, and confirms my suspicions about what was causing it.

The ability to merge databases automatically would be a handy feature for me, given it seems like the heavy lifting has already been done by the Merge database feature. However if it's not a common issue, it could be resolved by mentioning it in the docs when choosing a sync provider. https://keepassxc.org/docs/#faq-cloudsync

Really nice piece of software that keeps getting better. Thanks for all the effort 🙂

[jovandeginste]

I'll just chime in with my experience.

I used to use Dropbox for syncing, but switched to syncthing when Dropbox enforced their nr. of devices on Linux. My workflow is probably what triggers this; I have:

• a Linux home server which is almost always on
• an Android phone (always on and logged on) and several tablets (not usually logged on)
• a Linux desktop (work)
• a Linux laptop (work)
• a Windows laptop (gaming)

... which all share that one syncthing volume where my kdbx file is on, and where I use either KeepassXC (Linux/Windows) or Keepass2Android. I almost never edit the kdbx file on Android, I often (daily or more) edit it on the desktop or laptops.

Since I don't like waiting too much, I usually suspend/hibernate the desktop and laptops when I'm done using them. This is probably what triggers the merge problem.

My hunch:

Is it possible that KeepassXC saves an update to disk when it comes out of suspend mode (because time has passed), before syncthing had a chance of syncing?

[droidmonkey]

Make sure your syncthing is actually syncing your database when it is changed. Are you using always save after every change?

[jovandeginste]

I do

[jovandeginste]

I can share my settings, if you wish

[droidmonkey]

Most likely syncthing daemon not actually syncing in a timely manner. We don't hold a lock on files, as soon as it is written we release the file.

[jovandeginste]

I suddenly had a hunch and tested it. I realized the sync-issue started manifesting itself after I configured KeepassXC Secret Service Integration.

The options are rather limited, here are my settings:

[FdoSecrets]
Enabled=true
NoConfirmDeleteItem=false
ShowNotification=false

I think one of the programs using the Secret Service (probably Evolution) is updating a secret right after coming back from hibernation, which is then saved in KeepassXC, before syncthing had a change of syncing the remote changes back.

Not sure who to "blame" here...

On a tangential note, I don't really like this Secret Service thing too much, since it was a serious PITA to kill the default in Gnome and have KeepassXC reliably fulfil that role...

[droidmonkey]

Ok, neither of those problems are our fault.

[boernsen-development]

Just another attempt from my side to make this issue and possible solutions more obvious.

Issue:

• If using read-write cloud sync for a single database on multiple devices, which are likely to loose connection, conflict files are likely to appear without the user noticing it. This is very likely for decentral tools such as Syncthing in combination with multiple devices.
• These conflict files are likely to be overseen (thus passwords seem to suddenly disappear) and merging them into the actual database is very inconvenient.

Workarounds:

• Use a dedicated database for each device, which is edited only on that device (read-write) and merge changes from other devices manually while treating other devices' databases as read-only. (all databases are synced between the devices)
• Manually check for conflict files regularly and merge them manually. This requires to enter the same password again.

Some suggestions for development (collected from the discussions above):

• Mention the issue at https://keepassxc.org/docs/#faq-cloudsync
• Add automatic detection for conflict files. This should be fairly easy as they all contain the original filename and "conflict" (case-insensitive) somewhere in the filename. See my comment above for documentations of various sync providers.
• Add dialog noticing about existing conflict files or mention them on the unlock screen.
• Add feature to also open/unlock all related conflict databases when unlocking a database
• Add feature to re-use the password of an unlocked database when merging a locked database.
• Add feature to merge multiple databases at once.
• Add feature to optionally delete the merged databases (i.e. move them to bin).

I think it's not necessary to implement all of them. Maybe just the simplest ones would be enough to overcome the usability drawbacks of finding the conflict databases, merging them one by one and entering the same password again and again.
I hope this way we'll find some solutions or usability features that are simple enough to be picked for development one day.

Please let me know if you have more suggestions and I'll add them. Alternatively, the OP could also add such a list to the description if he wants to.

[erikvanoosten]

IMHO the simplest thing that can be implemented to overcome the usability problem are these 2 small features:

• Allow multiple select when selecting the file(s) for Merge Database, and then merge the selected databases one by one.
• When opening the to-be-merged database, only ask for a password if it can not be opened with the credentials of the current database. (Variations: 1. do show the password dialogue, but pre-fill the password, 2. display an 'are-you-sure' prompt.)

These features can be implemented independently, and they bring usability value independently as well.

[droidmonkey]

Good ideas!

[tobiasBora]

@erikvanoosten I would say that it is also important to automatically (this is important since most of the time people won't even realize there is a conflict) start the merge process whenever keepassxc sees that a new conflicting file has been created, potentially with a message:

Warning: keepassxc detected a conflict because the database was modified on another device. Do you want to automatically merge the two databases (X.kdbx and X_conflict.kbdx) to get rid of the conflict?
[Yes/No/Yes but do not remove the conflicted file/Yes but keep a backup of both files]

By default we could automatically search for conflict files using the syntax used by famous cloud providers, and provide to the user a function to add a regex to find conflicting files (I guess we can search for files in the current directory only, it is certainly quicker and safer). This regex could certainly be saved in the database, just see the below security concern in that case.

In term of security implication, I can only imagine one scenario (really special I guess) where storing the regex in the database could be a problem:

• if an adversary A is able to share a password database D with a party B
• if B stores another private database P in the same folder
• If B chooses exactly the same password for P and D

then A could learn the database P by adding the regex .* in D (or if B named its database with the word conflict in it… which is quite unlikely). Then P would be merged into D since they have the same password, and A would learn the passwords of B.

Of course, this attack is quite unlikely to happen in a real life since B has no reasons to use the same password for both databases and put them in the same folder… (with a regular cloud provider both files would have been shared to B which is already an issue). But anyway this "attack" can be mitigated by asking for a confirmation if the database uses a custom regex, while printing a warning if a database is created with a name containing the word conflict.

[dolfandringa]

IMHO the simplest thing that can be implemented to overcome the usability problem are these 2 small features:

* Allow _multiple select_ when selecting the file(s) for Merge Database, and then merge the selected databases one by one.

* When opening the to-be-merged database, _only ask for a password_ if it can not be opened with the credentials of the current database. (Variations: 1. do show the password dialogue, but pre-fill the password, 2. display an 'are-you-sure' prompt.)

These features can be implemented independently, and they bring usability value independently as well.

I agree with these solutions. They solve most. For me the last step would be to automate the merge and record the paths of the databases to auto-merge. That would allow me to open my local copy, and merge with paths in the FS when opening/saving the database, if those files are available.

[Prophet-Phoenix]

Long time user of KeePassXC here. Love it, thank you! ❤️
Also using syncthing for quite some time and i also have syncconflicts when making fast changes to the kdbx file that aren't picked up by syncthing fast enough and not synced to all devices. And to be clear i think it is just syncthings limitations and it's not keepassxc at fault.

What i am missing in merging kdbx files is the option to not import deleted entrys again.
So if i deleted an entry "github.com" on computer A and earlier changed an entry "google.com" on computer B (which was not synced to computer A because of syncthing), the deleted entry "github.com" would get merged back in A, when merging the conflicting file B, rigth? Coming from B, "github.com" is a new entry. All other merges would be made correctly judging by date of modification.

Maybe show new entrys with older modification or creation date and ask which ones to import?

[droidmonkey]

the deleted entry "github.com" would get merged back in A, when merging the conflicting file B, rigth?

It shouldn't. When you delete an entry, a record of that deletion is stored in the database file (the UUID of the entry). This is used to "re-delete" the entry if it is ever seen again on a merge. This is actually the root cause of #6477

[Prophet-Phoenix]

I tested again and turns out that my original kdbx file does not have a recycle bin, as the new ones have. I created a new database and merged with my original one and now it behaves like you say. So thanks!

Does the merge functionality rely on the recycle bin folder? It does feel a little odd that all my internet password history is in the recycle bin forever now. But i can live with that probably or delete them for good, as long as i dont have conflicting database files lying around.

[droidmonkey]

Recycle Bin doesn't mean the entry was deleted, it was just moved to that group. The deleted UUID only gets written when it is actually deleted. When you empty your recycle bin this will occur.

[serpro69]

I'm not sure I completely understand the problem with sync of multiple databases.
But maybe what I'm doing myself isn't actually doing what I expect it to?

Say I have two devices: A and B.

On each device, I have it's own .kdbx file. I do sync only in one direction, and then merge that with the "origin". For example:

• A has a.kdbx open and B has b.kdbx open locally
• A makes changes to a.kdbx and sends it to B
• B does merge of b.kdbx with a.kdbx
• B sends b.kdbx to A
• A does merge of a.kdbx with b.kdbx
• both A and B now have their own updated .kdbx files, which have identical contents

Am I doing something wrong? (apart from the fact that it does have a manual step to merge things, which I don't really much mind, at least at this point)

[droidmonkey]

@serpro69 that is certainly a way to do it that would avoid A and B conflicting with each other on writes (ie, they both make a change at nearly the same time before a sync can occur). However, if that is not possible because you are the only one making changes, then you are actually wasting your time. Just use one database and sync (send the file) between A and B.

[serpro69]

Thanks for the reply @droidmonkey !
No, I don't have situations where changes to both might occur at the same time. But the problem for me is that A and B aren't always running at the same time and hence changes might not always be synced right away. I use syncthing for file synchronization, so I need to actually power on both to sync when I need to. Until I can set-up a third device which would always run and through which others could sync, I don't see how I could always sync between A and B directly w/o accidentally loosing something.
Let's say I make a change on B, which isn't synced, and then I make a change on A which also isn't synced (and this I do have sometimes). When I do sync, wouldn't that result in a conflict because both files have diverged from a common point? I'm not sure how syncthing would resolve that, but I don't think that it would be too happy about it :D So my assumption is that in this case I would still need to fix this manually with a merge via keepassxc.

[droidmonkey]

You could setup a raspberry pi as a third party. Syncthing should create a conflict file, it does keep a version history anyway. You can always test with a dummy database.