Skip to content

Commit 26103b2

Browse files
committed
feat(connectors): add 9 knowledge base connectors
Box, Zoho Desk, PagerDuty, Trello, Microsoft Excel, Google Slides, Google Vault, Mintlify, and SFTP. Selected by intersecting the published connector catalogs of Glean, Onyx, Dust, Vectara, Writer, Guru, Elastic, Microsoft 365 Copilot, Notion AI, Unstructured, and Airbyte against services that already ship a Sim block, so OAuth providers, credentials, and icons are reused. Box was the largest gap, appearing in 7-8 of ~10 catalogs. Every connector was validated against live provider documentation twice, the second pass treating the first pass's conclusions as unproven. Notable correctness work that came out of that: Listing truncation. The sync engine hard-deletes documents past a cap that is not flagged with `listingCapped`, and five connectors had a path there — an empty Mintlify discovery, Zoho Desk's exact-multiple default caps, Trello's archived lists and 1000-card ceiling, a Google Vault cursor bailout, and a PagerDuty stalled page. The engine also gained a backstop: an empty or collapsed listing blocks deletion reconciliation until the same observation repeats on a consecutive sync, reconstructed from existing sync-log counters so no migration is needed. API alignment. `desk.zoho.ca` does not resolve (Canada is `desk.zohocloud.ca`, and Singapore and UAE were missing); `modifiedTime` is absent from Zoho's ticket list projection, so every ticket re-embedded on every sync; Trello's `dateLastActivity` is documented to miss some edits; PagerDuty's 10,000-record ceiling bounds `offset + limit`, not offset; Excel indexed dates as raw serial numbers while Google Sheets renders them; Google Vault truncated at roughly 249 matters. Security. SFTP followed symlinks in `getDocument` and composed unchecked server-supplied filenames into paths; it now also supports optional host-key fingerprint verification, which runs during key exchange before any password is sent. Trello interpolated user-supplied board ids into URL paths. Google Vault is narrowed to `ediscovery.readonly`. `getDataverseBaseUrl` accepted any host while attaching a bearer token, and is pinned to Microsoft's Dataverse domains — pre-existing shipped code, fixed here. Also adds `ConnectorAuthConfig.optional` so a public source can be configured without inventing an API key, and teaches the scope check that a granted read-write scope satisfies a required `.readonly` sibling. Microsoft Dataverse was built and then removed: its OAuth cannot complete consent. Dataverse requires a per-environment resource URI, the provider declares a static `https://dynamics.microsoft.com/user_impersonation` that is not an Entra Application ID URI, and the environment URL is only collected after the credential exists. That predates this change and also affects the 12 shipped Dataverse tools.
1 parent 3051954 commit 26103b2

41 files changed

Lines changed: 7416 additions & 31 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

apps/docs/content/docs/en/knowledgebase/connectors.mdx

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -14,23 +14,24 @@ Connectors continuously sync documents from external services into your knowledg
1414

1515
<Image src="/static/connectors/connectors-sources.png" alt="Connect Source picker showing a searchable list of available connectors including Airtable, Asana, Confluence, Discord, Dropbox, Evernote, Fireflies, GitHub, and Gmail" width={800} height={500} />
1616

17-
Sim ships with 49 built-in connectors:
17+
Sim ships with 61 built-in connectors:
1818

1919
| Category | Connectors |
2020
|----------|-----------|
21-
| **Productivity** | Notion, Confluence, Asana, Linear, Jira, Jira Service Management, Monday, Google Calendar, Google Sheets, Google Forms, Typeform |
22-
| **Cloud Storage** | Google Drive, Dropbox, OneDrive, SharePoint, Amazon S3 |
23-
| **Documents** | Google Docs, WordPress, Webflow, DocuSign |
21+
| **Productivity** | Notion, Confluence, Asana, Linear, Jira, Jira Service Management, Monday, Trello, ClickUp, Google Calendar, Google Sheets, Google Forms, Microsoft Excel, Typeform |
22+
| **Cloud Storage** | Google Drive, Dropbox, OneDrive, SharePoint, Box, Amazon S3, SFTP |
23+
| **Documents** | Google Docs, Google Slides, Mintlify, WordPress, Webflow, DocuSign |
2424
| **Development** | GitHub, GitLab, Azure DevOps, Sentry |
25-
| **Communication** | Slack, Discord, Microsoft Teams, Reddit, YouTube |
25+
| **Communication** | Slack, Discord, Microsoft Teams, Reddit, X, YouTube |
2626
| **Email** | Gmail, Outlook |
2727
| **CRM** | HubSpot, Salesforce |
28-
| **Support** | Intercom, ServiceNow, Zendesk |
29-
| **Incident Management** | incident.io, Rootly |
28+
| **Support** | Intercom, ServiceNow, Zendesk, Zoho Desk |
29+
| **Incident Management** | incident.io, Rootly, PagerDuty |
3030
| **Data** | Airtable |
3131
| **Note-taking** | Evernote, Obsidian |
32-
| **Meetings** | Zoom, Gong, Grain, Granola, Fathom, Fireflies |
32+
| **Meetings** | Zoom, Google Meet, Gong, Grain, Granola, Fathom, Fireflies |
3333
| **Recruiting** | Greenhouse, Ashby |
34+
| **Compliance** | Google Vault |
3435

3536
## Adding a Connector
3637

@@ -55,6 +56,9 @@ Other connectors use **API keys** or **personal access tokens** instead. The set
5556
| **YouTube** | YouTube Data API key from the Google Cloud Console |
5657
| **Amazon S3** | Secret Access Key (the Access Key ID, region, and bucket are entered as config fields) |
5758
| **Sentry** | Auth token with `project:read` and `event:read` scopes |
59+
| **PagerDuty** | REST API key from Integrations → API Access Keys |
60+
| **SFTP** | Password or unencrypted private key (host, port, username, and root path are entered as config fields) |
61+
| **Mintlify** | API key — optional for public documentation sites, which sync from `llms.txt` |
5862

5963
<Callout type="info">
6064
If you rotate an API key in the external service, update it in Sim as well — OAuth tokens refresh automatically, but API keys do not.

apps/sim/app/api/tools/sftp/utils.ts

Lines changed: 104 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,13 @@
1+
import { createHash } from 'node:crypto'
2+
import { createLogger } from '@sim/logger'
3+
import { safeCompare } from '@sim/security/compare'
14
import { toError } from '@sim/utils/errors'
25
import { type Attributes, Client, type ConnectConfig, type SFTPWrapper } from 'ssh2'
36
import { validateDatabaseHost } from '@/lib/core/security/input-validation.server'
47
import { readNodeStreamToBufferWithLimit } from '@/lib/core/utils/stream-limits'
58

9+
const logger = createLogger('SftpUtils')
10+
611
const S_IFMT = 0o170000
712
const S_IFDIR = 0o040000
813
const S_IFREG = 0o100000
@@ -15,9 +20,44 @@ export interface SftpConnectionConfig {
1520
password?: string | null
1621
privateKey?: string | null
1722
passphrase?: string | null
23+
/**
24+
* Idle socket timeout in ms, forwarded to ssh2's `sock.setTimeout`. Left
25+
* unset the socket has no idle timeout at all (ssh2 defaults it to `0`).
26+
*/
1827
timeout?: number
1928
keepaliveInterval?: number
2029
readyTimeout?: number
30+
/**
31+
* Expected SHA-256 host key fingerprint in the format `ssh-keyscan` and
32+
* OpenSSH print (`SHA256:<base64>`). The `SHA256:` prefix and any base64
33+
* padding are optional. When set, a server presenting a different host key is
34+
* rejected before authentication runs. When omitted, the host is not
35+
* verified — ssh2's default behavior.
36+
*/
37+
hostFingerprint?: string | null
38+
}
39+
40+
/**
41+
* Normalizes a user-supplied SHA-256 fingerprint for comparison: trims, drops
42+
* an optional `SHA256:` prefix, and strips base64 `=` padding, which OpenSSH
43+
* omits but copy/paste sources sometimes include.
44+
*/
45+
function normalizeSha256Fingerprint(value: string): string {
46+
return value
47+
.trim()
48+
.replace(/^sha256:/i, '')
49+
.replace(/=+$/, '')
50+
.trim()
51+
}
52+
53+
/**
54+
* Computes the OpenSSH SHA-256 fingerprint of a host key. ssh2 hands the
55+
* verifier the raw SSH wire-format public key blob — the same bytes OpenSSH
56+
* base64-encodes into `known_hosts` — so hashing it directly reproduces the
57+
* unpadded base64 digest that `ssh-keyscan | ssh-keygen -lf -` prints.
58+
*/
59+
function computeHostKeyFingerprint(hostKey: Buffer): string {
60+
return createHash('sha256').update(hostKey).digest('base64').replace(/=+$/, '')
2161
}
2262

2363
/**
@@ -93,6 +133,11 @@ function formatSftpError(err: Error, config: { host: string; port: number }): Er
93133
/**
94134
* Creates an SSH connection for SFTP using the provided configuration.
95135
* Uses ssh2 library defaults which align with OpenSSH standards.
136+
*
137+
* When `hostFingerprint` is supplied the server's host key is pinned to it and
138+
* a mismatch aborts the handshake before any credential is sent. Without it
139+
* ssh2 accepts whatever host key answers, which is the pre-existing behavior
140+
* kept for backward compatibility.
96141
*/
97142
export async function createSftpConnection(config: SftpConnectionConfig): Promise<Client> {
98143
const host = config.host
@@ -132,6 +177,50 @@ export async function createSftpConnection(config: SftpConnectionConfig): Promis
132177
if (config.keepaliveInterval !== undefined) {
133178
connectConfig.keepaliveInterval = config.keepaliveInterval
134179
}
180+
if (config.timeout !== undefined) {
181+
connectConfig.timeout = config.timeout
182+
}
183+
184+
const suppliedFingerprint = config.hostFingerprint?.trim()
185+
const expectedFingerprint = suppliedFingerprint
186+
? normalizeSha256Fingerprint(suppliedFingerprint)
187+
: undefined
188+
189+
/**
190+
* Fail closed rather than silently skipping verification. A value that is
191+
* non-blank but normalizes away (`SHA256:`, `=`) would otherwise leave no
192+
* `hostVerifier` installed, trusting whatever host answers — the opposite
193+
* of what supplying a fingerprint asks for.
194+
*/
195+
if (suppliedFingerprint && !expectedFingerprint) {
196+
throw new Error(
197+
'Host key fingerprint is not a valid SHA-256 fingerprint. Expected the base64 form printed by `ssh-keyscan <host> | ssh-keygen -lf -`.'
198+
)
199+
}
200+
201+
/**
202+
* Set when the pinned fingerprint does not match. ssh2 reports the
203+
* rejection through a generic `'error'` event, so the precise cause is
204+
* carried out of the verifier rather than re-derived from that message.
205+
*/
206+
let hostKeyRejection: Error | undefined
207+
208+
if (expectedFingerprint) {
209+
connectConfig.hostVerifier = (hostKey: Buffer): boolean => {
210+
const actualFingerprint = computeHostKeyFingerprint(hostKey)
211+
if (safeCompare(actualFingerprint, expectedFingerprint)) {
212+
return true
213+
}
214+
hostKeyRejection = new Error(
215+
`Host key verification failed for ${host}:${port}. ` +
216+
`Expected SHA256:${expectedFingerprint} but the server presented SHA256:${actualFingerprint}. ` +
217+
`Either the server's host key changed, or the connection was intercepted. ` +
218+
`Re-run "ssh-keyscan -t rsa,ecdsa,ed25519 ${host}" to confirm the current key before updating the fingerprint.`
219+
)
220+
logger.warn('SFTP host key fingerprint mismatch', { host, port })
221+
return false
222+
}
223+
}
135224

136225
if (hasPrivateKey) {
137226
connectConfig.privateKey = config.privateKey!
@@ -147,7 +236,21 @@ export async function createSftpConnection(config: SftpConnectionConfig): Promis
147236
})
148237

149238
client.on('error', (err) => {
150-
reject(formatSftpError(err, { host, port }))
239+
reject(hostKeyRejection ?? formatSftpError(err, { host, port }))
240+
})
241+
242+
/**
243+
* ssh2 only re-emits the socket's `'timeout'` event; it never destroys the
244+
* socket, so without this the connection would sit open forever after the
245+
* idle timeout elapsed.
246+
*/
247+
client.on('timeout', () => {
248+
client.destroy()
249+
reject(
250+
new Error(
251+
`Connection to ${host}:${port} timed out after ${config.timeout}ms of inactivity.`
252+
)
253+
)
151254
})
152255

153256
try {

apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/add-connector-modal/add-connector-modal.tsx

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,9 @@ export function AddConnectorModal({
8282

8383
const connectorConfig = selectedType ? CONNECTOR_META_REGISTRY[selectedType] : null
8484
const isApiKeyMode = connectorConfig?.auth.mode === 'apiKey'
85+
/** True when the connector declares its key optional (public sources need none). */
86+
const isApiKeyOptional =
87+
connectorConfig?.auth.mode === 'apiKey' && connectorConfig.auth.optional === true
8588
const connectorProviderId = useMemo(
8689
() =>
8790
connectorConfig && connectorConfig.auth.mode === 'oauth'
@@ -160,7 +163,7 @@ export function AddConnectorModal({
160163
const canSubmit = useMemo(() => {
161164
if (!connectorConfig) return false
162165
if (isApiKeyMode) {
163-
if (!apiKeyValue.trim()) return false
166+
if (!isApiKeyOptional && !apiKeyValue.trim()) return false
164167
} else {
165168
if (!effectiveCredentialId) return false
166169
}
@@ -174,6 +177,7 @@ export function AddConnectorModal({
174177
}, [
175178
connectorConfig,
176179
isApiKeyMode,
180+
isApiKeyOptional,
177181
apiKeyValue,
178182
effectiveCredentialId,
179183
isFieldVisible,
@@ -207,7 +211,11 @@ export function AddConnectorModal({
207211
{
208212
knowledgeBaseId,
209213
connectorType: selectedType,
210-
...(isApiKeyMode ? { apiKey: apiKeyValue } : { credentialId: effectiveCredentialId! }),
214+
...(isApiKeyMode
215+
? apiKeyValue.trim()
216+
? { apiKey: apiKeyValue }
217+
: {}
218+
: { credentialId: effectiveCredentialId! }),
211219
sourceConfig: finalSourceConfig,
212220
syncIntervalMinutes: syncInterval,
213221
},

0 commit comments

Comments
 (0)