serve s3: add --meta-db flag to persist object metadata using bbolt#9243
Open
danielloader wants to merge 3 commits intorclone:masterfrom
Open
serve s3: add --meta-db flag to persist object metadata using bbolt#9243danielloader wants to merge 3 commits intorclone:masterfrom
danielloader wants to merge 3 commits intorclone:masterfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of this change?
Ability to persist s3 object metadata when using serve s3 using bbolt, while retaining the in memory sync.Map behaviour as default.
Given bbolt was already being used in the project opted to use it for some simple key-value storing, and since afaik s3 object metadata is immutable after PUT I've opted to store it as a JSON value in the key rather than a more complex /bucket/key/metadata-key=value format.
Opted to use bbolt's native bucketing to isolate s3 buckets (directories when serving) to make deletion of the metadata simpler when deleting a bucket.
Kept the change as small as is possible with my skill set to retain the existing functionality as a default without refactoring in places where it's not necessary.
Been using rclone for 10 years now so figured I'd give contribution a go.
Was the change discussed in an issue or in the forum before?
No, was just looking to look for interest in the functionality speculatively as it suited my needs.
Checklist