feat(backend): add Bitbucket Server (Data Center) permission syncing

CHANGELOG.md

@@ -15,6 +15,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added `wa_askgh_login_wall_prompted` PostHog event fired when an unauthenticated user attempts to ask a question on Ask GitHub. [#933](https://github.com/sourcebot-dev/sourcebot/pull/933)
 - Added Bitbucket Server (Data Center) OAuth 2.0 SSO identity provider support (`provider: "bitbucket-server"`). [#934](https://github.com/sourcebot-dev/sourcebot/pull/934)
 - Added Bitbucket Server (Data Center) sync all repositories support. [#927](https://github.com/sourcebot-dev/sourcebot/pull/927)
+- Added permission syncing support for Bitbucket Server (Data Center), including account-driven and repo-driven sync. [#938](https://github.com/sourcebot-dev/sourcebot/pull/938)
 
 ### Changed
 - Hide version upgrade toast for askgithub deployment (`EXPERIMENT_ASK_GH_ENABLED`). [#931](https://github.com/sourcebot-dev/sourcebot/pull/931)

docs/docs/configuration/idp.mdx

@@ -74,9 +74,11 @@ in the GitHub identity provider config.
                 - `"Metadata" repository permissions (read)` (only needed if using permission syncing)
         </Tab>
         <Tab title="GitHub OAuth App">
-            Follow [this guide](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app) by GitHub to create an OAuth App. 
-            
+            Follow [this guide](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app) by GitHub to create an OAuth App.
+
             When asked to provide a callback url, provide `<sourcebot_url>/api/auth/callback/github` (ex. https://sourcebot.coolcorp.com/api/auth/callback/github)
+
+            If [permission syncing](/docs/features/permission-syncing#github) is enabled, also enable the `repo` scope.
         </Tab>
         </Tabs>
     </Step>
@@ -128,7 +130,7 @@ in the GitLab identity provider config.
         When configuring your application:
         - Set the callback URL to `<sourcebot_url>/api/auth/callback/gitlab` (ex. https://sourcebot.coolcorp.com/api/auth/callback/gitlab)
         - Enable the `read_user` scope
-        - If using for permission syncing, also enable the `read_api` scope
+        - If [permission syncing](/docs/features/permission-syncing#gitlab) is enabled, also enable the `read_api` scope
 
         The result of registering an OAuth application is an `APPLICATION_ID` (`CLIENT_ID`) and `SECRET` (`CLIENT_SECRET`) which you'll provide to Sourcebot.
     </Step>
@@ -182,7 +184,7 @@ in the Bitbucket Cloud identity provider config.
         When configuring your consumer:
         - Set the callback URL to `<sourcebot_url>/api/auth/callback/bitbucket-cloud` (ex. https://sourcebot.coolcorp.com/api/auth/callback/bitbucket-cloud)
         - Enable **Account: Read**
-        - If using for permission syncing, also enable **Repositories: Read**
+        - If [permission syncing](/docs/features/permission-syncing#bitbucket-cloud) is enabled, also enable **Repositories: Read**
 
         The result of creating an OAuth consumer is a `Key` (`CLIENT_ID`) and `Secret` (`CLIENT_SECRET`) which you'll provide to Sourcebot.
     </Step>
@@ -220,7 +222,8 @@ in the Bitbucket Cloud identity provider config.
 
 ### Bitbucket Server
 
-A Bitbucket Server (Data Center) connection can be used for [authentication](/docs/configuration/auth).
+A Bitbucket Server (Data Center) connection can be used for [authentication](/docs/configuration/auth) and/or [permission syncing](/docs/features/permission-syncing). This is controlled using the `purpose` field
+in the Bitbucket Server identity provider config.
 
 <Accordion title="instructions">
 <Steps>
@@ -231,6 +234,7 @@ A Bitbucket Server (Data Center) connection can be used for [authentication](/do
 
         When configuring your application:
         - Set the redirect URL to `<sourcebot_url>/api/auth/callback/bitbucket-server` (ex. https://sourcebot.coolcorp.com/api/auth/callback/bitbucket-server)
+        - If [permission syncing](/docs/features/permission-syncing#bitbucket-data-center) is enabled, also enable the `REPO_READ` scope
 
         The result of creating the application is a `CLIENT_ID` and `CLIENT_SECRET` which you'll provide to Sourcebot.
     </Step>
@@ -247,7 +251,10 @@ A Bitbucket Server (Data Center) connection can be used for [authentication](/do
             "identityProviders": [
                 {
                     "provider": "bitbucket-server",
-                    "purpose": "sso",
+                    // "sso" for auth + perm sync, "account_linking" for only perm sync
+                    "purpose": "account_linking",
+                    // if purpose == "account_linking" this controls if a user must connect to the IdP
+                    "accountLinkingRequired": true,
                     "baseUrl": "https://bitbucket.example.com",
                     "clientId": {
                         "env": "YOUR_CLIENT_ID_ENV_VAR"

docs/docs/connections/bitbucket-data-center.mdx

@@ -80,35 +80,75 @@ If you're not familiar with Sourcebot [connections](/docs/connections/overview),
 
 ## Authenticating with Bitbucket Data Center
 
-In order to index private repositories, you'll need to provide an access token to Sourcebot via a [token](/docs/configuration/config-file#tokens).
-
-Create an access token for the desired scope (repo, project, or workspace). Visit the official [Bitbucket Data Center docs](https://confluence.atlassian.com/bitbucketserver/http-access-tokens-939515499.html)
-for more info.
-
-1. Add the `token` property to your connection config:
-
-```json
-{
-    "type": "bitbucket",
-    "deploymentType": "server",
-    "url": "https://mybitbucketdeployment.com",
-    "token": {
-        // note: this env var can be named anything. It
-        // doesn't need to be `BITBUCKET_TOKEN`.
-        "env": "BITBUCKET_TOKEN"
-    }
-    // .. rest of config ..
-}
-```
-
-2. Pass this environment variable each time you run Sourcebot:
-
-```bash
-docker run \
-    -e BITBUCKET_TOKEN=<ACCESS_TOKEN> \
-    /* additional args */ \
-    ghcr.io/sourcebot-dev/sourcebot:latest
-```
+In order to index private repositories, you'll need to provide a [HTTP Access Token](https://confluence.atlassian.com/bitbucketserver/http-access-tokens-939515499.html). Tokens can be scoped to a user account, a project, or an individual repository. Only repositories visible to the token will be able to be indexed by Sourcebot.
+
+<Note>
+  If [permission syncing](/docs/features/permission-syncing#bitbucket-data-center) is enabled, the token must have **Repository Admin** permissions so Sourcebot can read repository-level user permissions.
+</Note>
+
+<Tabs>
+    <Tab title="User account token">
+        User account tokens grant access to all repositories the user can access. Because these are tied to a specific user account, you must also set the `user` field to that user's username.
+
+        1. In Bitbucket Data Center, navigate to your profile → **Manage account** → **HTTP access tokens** and click **Create token**. Give it a name and grant it **Project read** and **Repository read** permissions.
+
+        2. Add the `user` (your Bitbucket username) and `token` properties to your connection config:
+
+        ```json
+        {
+            "type": "bitbucket",
+            "deploymentType": "server",
+            "url": "https://mybitbucketdeployment.com",
+            "user": "myusername",
+            "token": {
+                // note: this env var can be named anything. It
+                // doesn't need to be `BITBUCKET_TOKEN`.
+                "env": "BITBUCKET_TOKEN"
+            }
+            // .. rest of config ..
+        }
+        ```
+
+        3. Pass this environment variable each time you run Sourcebot:
+
+        ```bash
+        docker run \
+            -e BITBUCKET_TOKEN=<ACCESS_TOKEN> \
+            /* additional args */ \
+            ghcr.io/sourcebot-dev/sourcebot:latest
+        ```
+    </Tab>
+    <Tab title="Project / repository token">
+        Project and repository tokens are scoped to a specific project or repository.
+
+        1. In Bitbucket Data Center, navigate to the project or repository → **Settings** → **HTTP access tokens** and click **Create token**. Give it a name and grant it **Repository read** and **Project read** permissions.
+
+        2. Add the `token` property to your connection config:
+
+        ```json
+        {
+            "type": "bitbucket",
+            "deploymentType": "server",
+            "url": "https://mybitbucketdeployment.com",
+            "token": {
+                // note: this env var can be named anything. It
+                // doesn't need to be `BITBUCKET_TOKEN`.
+                "env": "BITBUCKET_TOKEN"
+            }
+            // .. rest of config ..
+        }
+        ```
+
+        3. Pass this environment variable each time you run Sourcebot:
+
+        ```bash
+        docker run \
+            -e BITBUCKET_TOKEN=<ACCESS_TOKEN> \
+            /* additional args */ \
+            ghcr.io/sourcebot-dev/sourcebot:latest
+        ```
+    </Tab>
+</Tabs>
 
 ## Troubleshooting
 If you're seeing errors like `TypeError: fetch failed` when fetching repo info, it may be that Sourcebot is refusing to connect to your self-hosted Bitbucket instance due to unrecognized SSL certs. Try setting the `NODE_TLS_REJECT_UNAUTHORIZED=0` environment variable or providing Sourcebot your certs through the `NODE_EXTRA_CA_CERTS` environment variable.

docs/docs/features/permission-syncing.mdx

@@ -40,7 +40,7 @@ We are actively working on supporting more code hosts. If you'd like to see a sp
 | [GitHub (GHEC & GHEC Server)](/docs/features/permission-syncing#github) | ✅ |
 | [GitLab (Self-managed & Cloud)](/docs/features/permission-syncing#gitlab) | ✅ |
 | [Bitbucket Cloud](/docs/features/permission-syncing#bitbucket-cloud) | 🟠 Partial |
-| Bitbucket Data Center | 🛑 |
+| [Bitbucket Data Center](/docs/features/permission-syncing#bitbucket-data-center) | 🟠 Partial |
 | Gitea | 🛑 |
 | Gerrit | 🛑 |
 | Generic git host | 🛑 |
@@ -50,7 +50,8 @@ We are actively working on supporting more code hosts. If you'd like to see a sp
 ## GitHub
 
 Prerequisites:
-- Configure GitHub as an [external identity provider](/docs/configuration/idp).
+- Configure a [GitHub connection](/docs/connections/github).
+- Configure GitHub as an [external identity provider](/docs/configuration/idp#github).
 - **If you are using a self-hosted GitHub instance**, you must also set the `baseUrl` property of the `github` identity provider in the [config file](/docs/configuration/config-file) to the base URL of your GitHub instance (e.g. `https://github.example.com`).
 
 Permission syncing works with **GitHub.com**, **GitHub Enterprise Cloud**, and **GitHub Enterprise Server**. For organization-owned repositories, users that have **read-only** access (or above) via the following methods will have their access synced to Sourcebot:
@@ -61,27 +62,29 @@ Permission syncing works with **GitHub.com**, **GitHub Enterprise Cloud**, and *
 - Organization owners.
 
 **Notes:**
-- A GitHub [external identity provider](/docs/configuration/idp) must be configured to (1) correlate a Sourcebot user with a GitHub user, and (2) to list repositories that the user has access to for [User driven syncing](/docs/features/permission-syncing#how-it-works).
+- A GitHub [external identity provider](/docs/configuration/idp#github) must be configured to (1) correlate a Sourcebot user with a GitHub user, and (2) to list repositories that the user has access to for [User driven syncing](/docs/features/permission-syncing#how-it-works).
 - OAuth tokens must assume the `repo` scope in order to use the [List repositories for the authenticated user API](https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#list-repositories-for-the-authenticated-user) during [User driven syncing](/docs/features/permission-syncing#how-it-works). Sourcebot **will only** use this token for **reads**.
 
 ## GitLab
 
 Prerequisites:
-- Configure GitLab as an [external identity provider](/docs/configuration/idp).
+- Configure a [GitLab connection](/docs/connections/gitlab).
+- Configure GitLab as an [external identity provider](/docs/configuration/idp#gitlab).
 - **If you are using a self-hosted GitLab instance**, you must also set the `baseUrl` property of the `gitlab` identity provider in the [config file](/docs/configuration/config-file) to the base URL of your GitLab instance (e.g. `https://gitlab.example.com`).
 
 Permission syncing works with **GitLab Self-managed** and **GitLab Cloud**. Users with **Guest** role or above with membership to a group or project will have their access synced to Sourcebot. Both direct and indirect membership to a group or project will be synced with Sourcebot. For more details, see the [GitLab docs](https://docs.gitlab.com/user/project/members/#membership-types).
 
 
 **Notes:**
-- A GitLab [external identity provider](/docs/configuration/idp) must be configured to (1) correlate a Sourcebot user with a GitLab user, and (2) to list repositories that the user has access to for [User driven syncing](/docs/features/permission-syncing#how-it-works).
+- A GitLab [external identity provider](/docs/configuration/idp#gitlab) must be configured to (1) correlate a Sourcebot user with a GitLab user, and (2) to list repositories that the user has access to for [User driven syncing](/docs/features/permission-syncing#how-it-works).
 - OAuth tokens require the `read_api` scope in order to use the [List projects for the authenticated user API](https://docs.gitlab.com/ee/api/projects.html#list-all-projects) during [User driven syncing](/docs/features/permission-syncing#how-it-works).
 - [Internal GitLab projects](https://docs.gitlab.com/user/public_access/#internal-projects-and-groups) are **not** enforced by permission syncing and therefore are visible to all users. Only [private projects](https://docs.gitlab.com/user/public_access/#private-projects-and-groups) are enforced.
 
 ## Bitbucket Cloud
 
 Prerequisites:
-- Configure Bitbucket Cloud as an [external identity provider](/docs/configuration/idp).
+- Configure a [Bitbucket Cloud connection](/docs/connections/bitbucket-cloud).
+- Configure Bitbucket Cloud as an [external identity provider](/docs/configuration/idp#bitbucket-cloud).
 
 Permission syncing works with **Bitbucket Cloud**. OAuth tokens must assume the `account` and `repository` scopes.
 
@@ -98,9 +101,33 @@ If your workspace relies heavily on group or project-level permissions rather th
 </Warning>
 
 **Notes:**
-- A Bitbucket Cloud [external identity provider](/docs/configuration/idp) must be configured to (1) correlate a Sourcebot user with a Bitbucket Cloud user, and (2) to list repositories that the user has access to for [User driven syncing](/docs/features/permission-syncing#how-it-works).
+- A Bitbucket Cloud [external identity provider](/docs/configuration/idp#bitbucket-cloud) must be configured to (1) correlate a Sourcebot user with a Bitbucket Cloud user, and (2) to list repositories that the user has access to for [User driven syncing](/docs/features/permission-syncing#how-it-works).
 - OAuth tokens require the `account` and `repository` scopes. The `repository` scope is required to list private repositories during [User driven syncing](/docs/features/permission-syncing#how-it-works).
 
+## Bitbucket Data Center
+
+Prerequisites:
+- Configure a [Bitbucket Data Center connection](/docs/connections/bitbucket-data-center).
+- Configure Bitbucket Data Center as an [external identity provider](/docs/configuration/idp#bitbucket-server).
+
+Permission syncing works with **Bitbucket Data Center**. OAuth tokens must assume the `PUBLIC_REPOS` and `REPO_READ` scopes.
+
+<Warning>
+**Partial coverage for repo-driven syncing.** Repo-driven syncing only captures users who have been **directly and explicitly** granted access to the repository. Users who have access via any of the following are **not** captured by repo-driven syncing:
+
+- Project-level permissions (inherited by all repos in the project)
+- Group membership
+
+These users **will** still gain access via [user-driven syncing](/docs/features/permission-syncing#how-it-works), which fetches all repositories accessible to each authenticated user using the `REPO_READ` scope. However, there may be a delay between when access is granted and when affected users see the repository in Sourcebot (up to the `experiment_userDrivenPermissionSyncIntervalMs` interval, which defaults to 24 hours).
+
+If your instance relies heavily on project or group-level permissions, we recommend reducing the `experiment_userDrivenPermissionSyncIntervalMs` interval to limit the window of delay.
+</Warning>
+
+**Notes:**
+- A Bitbucket Data Center [external identity provider](/docs/configuration/idp#bitbucket-server) must be configured to (1) correlate a Sourcebot user with a Bitbucket Data Center user, and (2) to list repositories that the user has access to for [User driven syncing](/docs/features/permission-syncing#how-it-works).
+- The connection token must have **Repository Read** permissions so Sourcebot can read repository-level user permissions for [Repo driven syncing](/docs/features/permission-syncing#how-it-works).
+- OAuth tokens require the `REPO_READ` scope to list accessible repositories during [User driven syncing](/docs/features/permission-syncing#how-it-works).
+
 # How it works
 
 Permission syncing works by periodically syncing ACLs from the code host(s) to Sourcebot to build an internal mapping between Users and Repositories. This mapping is hydrated in two directions:

docs/snippets/schemas/v3/identityProvider.schema.mdx

@@ -850,7 +850,10 @@
           "const": "bitbucket-server"
         },
         "purpose": {
-          "const": "sso"
+          "enum": [
+            "sso",
+            "account_linking"
+          ]
         },
         "clientId": {
           "anyOf": [
@@ -919,6 +922,10 @@
             "https://bitbucket.example.com"
           ],
           "pattern": "^https?:\\/\\/[^\\s/$.?#].[^\\s]*$"
+        },
+        "accountLinkingRequired": {
+          "type": "boolean",
+          "default": false
         }
       },
       "required": [
@@ -1777,7 +1784,10 @@
           "const": "bitbucket-server"
         },
         "purpose": {
-          "const": "sso"
+          "enum": [
+            "sso",
+            "account_linking"
+          ]
         },
         "clientId": {
           "anyOf": [
@@ -1846,6 +1856,10 @@
             "https://bitbucket.example.com"
           ],
           "pattern": "^https?:\\/\\/[^\\s/$.?#].[^\\s]*$"
+        },
+        "accountLinkingRequired": {
+          "type": "boolean",
+          "default": false
         }
       },
       "required": [

docs/snippets/schemas/v3/index.schema.mdx

@@ -5395,7 +5395,10 @@
                 "const": "bitbucket-server"
               },
               "purpose": {
-                "const": "sso"
+                "enum": [
+                  "sso",
+                  "account_linking"
+                ]
               },
               "clientId": {
                 "anyOf": [
@@ -5464,6 +5467,10 @@
                   "https://bitbucket.example.com"
                 ],
                 "pattern": "^https?:\\/\\/[^\\s/$.?#].[^\\s]*$"
+              },
+              "accountLinkingRequired": {
+                "type": "boolean",
+                "default": false
               }
             },
             "required": [
@@ -6322,7 +6329,10 @@
                 "const": "bitbucket-server"
               },
               "purpose": {
-                "const": "sso"
+                "enum": [
+                  "sso",
+                  "account_linking"
+                ]
               },
               "clientId": {
                 "anyOf": [
@@ -6391,6 +6401,10 @@
                   "https://bitbucket.example.com"
                 ],
                 "pattern": "^https?:\\/\\/[^\\s/$.?#].[^\\s]*$"
+              },
+              "accountLinkingRequired": {
+                "type": "boolean",
+                "default": false
               }
             },
             "required": [