feat: Document management revamp, connector UX improvements, batch embeddings, and CI/DevOps enhancements#868
Conversation
…ment-adapter-class
- Updated the `resetBatchGate` function to accept a parameter for immediate resolution, enhancing batch processing control. - Changed the return type of the `useQuery` hook in `useComments` to specify `GetCommentsResponse`, improving type safety and clarity. - Adjusted the initial batch gate invocation to resolve immediately, streamlining the comments fetching process.
…-out functionality - Replaced the existing announcement card and empty state components with imports from the new announcements directory. - Introduced state management for the announcements sidebar in the LayoutDataProvider. - Updated navigation items to handle announcements sidebar toggling. - Integrated AnnouncementsSidebar into the LayoutShell for both mobile and desktop views. - Enhanced sidebar handling to improve user experience when navigating announcements.
add configurable extension backend url and login support
… job execution and integrating Docker metadata action for improved tagging and manifest creation
…and improve tagging logic for better handling of version tags
… formatting and clarity
…nd eliminate unnecessary exclusions
…ps and implementing a new DocumentsSidebar for better document management
…connector dialog comment in dashboard layout
…rts and buttons, and streamline type toggle functionality
…ashboard layout by eliminating unused component references
…actions and improve document tooltip display
…ntsTableShell with infinite scroll functionality for improved document loading
…d update DocumentsSidebar for improved search functionality
…dapter-class refactor: Migrate document reindexing to `UploadDocumentAdapter` with unified indexing pipeline
Feat: added Announcements sidebar when logged in
… DocumentsSidebar for improved search and pagination handling
…ll by adding separate scroll references for desktop and mobile views
…buttons and enhancing search input functionality, while updating DocumentsSidebar for improved spacing
… by replacing icons for improved clarity and optimizing loading state management in useDocuments hook
…s by removing unused column visibility state and optimizing document loading logic in useDocuments hook
…s for current sort state, enhancing performance and clarity
…te UI components for improved layout and user experience
…nsistency across connector forms
…ight and enable rich colors
…mposio-google-drive-connector [Perf] Batch embedding, non-blocking search, chunks index & Google Drive UX fix
…urces and Report; update report panel handling in dashboard chat page for improved user experience
… width adjustments; refactor ReportPanelContent to streamline loading state handling
…ges; implement version tagging and multi-platform support for backend and web services
…mations and layout handling; integrate motion effects for smoother transitions
…r components; adjust sidebar behavior based on device type and improve right panel handling
fix: enhance docker build CI pipeline, update docker ports & docker docs
…ndling - Add check_permission to drive-picker-token endpoint (IDOR fix) - Use get_composio_service singleton + asyncio.to_thread to avoid blocking the event loop - Sanitize error detail in 500 response to prevent internal info leakage - Dispose picker on unmount to prevent orphaned overlay - Surface error state on Google Picker Action.ERROR instead of silently closing
Resolve 3 conflicts: - connector-popup.tsx: keep Picker modal coordination (pickerOpen state) - google-drive-config.tsx: keep Google Picker, discard upstream folder tree - composio-drive-config.tsx: accept upstream collapsible folder tree UX
[Improvement] UX for connectors: Google drive Picker, auto-index with default configs
- Added endpoint to list agent tools with metadata, excluding hidden tools. - Updated NewChatRequest and RegenerateRequest schemas to include disabled tools. - Integrated disabled tools management in the NewChatPage and Composer components. - Improved tool instructions and visibility in the system prompt. - Refactored tool registration to support hidden tools and default enabled states. - Enhanced document chunk creation to handle strict zip behavior. - Cleaned up imports and formatting across various files for consistency.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A major release bringing a fully revamped document management experience, significant connector UX improvements, backend performance gains through batch embeddings, and strengthened CI/DevOps pipelines.
Document Management Revamp
useDocumentSearchhookConnectToolsBannercomponent and connector tools strip in the sidebarConnector & Integration Improvements
drive-picker-tokenendpoint withGOOGLE_PICKER_API_KEYconfigBackend Performance & Features
embed_texts) with conditional strategy — batch for API providers, sequential for local modelsasyncio.to_threadforembed_textin FastAPI paths to avoid blocking the event loopchunks.document_idfor faster search retrievalUploadDocumentAdapterrefactor for file indexing and reindexinglast_logincolumn to user table for login trackingFrontend UI/UX
PencilIcon→Pen, updated lucide-react to v0.577.0)DocumentsTable,DashboardBreadcrumb,ChatHeader(from Thread)Inbox & Notifications
LayoutDataProviderCI/DevOps
.dockerignoreto streamline build processDocumentation
Test Plan
High-level PR Summary
This major release delivers a redesigned document management experience with docked sidebar mode, infinite scroll, and integrated search, alongside significant connector UX improvements including native Google Picker integration and auto-indexing after OAuth. The backend introduces batch embeddings for performance gains (conditional strategy for API vs local models), adds missing database indexes for faster search, and implements durable background deletion for documents and search spaces via Celery. The frontend enhances collaboration with real-time document mentions in chat (both inline chips and sidebar selections), multi-format report exports (PDF, DOCX, HTML, LaTeX, EPUB, ODT, plain text), and a new user settings page with profile management and API key configuration. CI/DevOps improvements include comprehensive GitHub Actions workflows for backend testing, enhanced Docker build with digest-based multi-arch manifests, and upgraded tooling (Biome v2 with Tailwind CSS parser). Notification filtering is expanded with category-based and source-type filters, plus a batched unread counts endpoint to reduce API calls. Gmail HTML rendering is improved by unwrapping layout tables, and YouTube playlist resolution is added for document collection. The agent system prompt is now scoped to only enabled tools, with explicit messaging about disabled capabilities.
⏱️ Estimated Review Time: 3+ hours
💡 Review Order Suggestion
.github/workflows/backend-tests.yml.github/workflows/docker-build.ymldocker/.env.exampledocker/docker-compose.ymldocker/docker-compose.dev.ymldocker/scripts/install.shdocker/scripts/install.ps1surfsense_backend/alembic/versions/103_add_last_login_to_user.pysurfsense_backend/alembic/versions/104_add_notification_composite_indexes.pysurfsense_backend/alembic/versions/105_add_chunks_document_id_index.pysurfsense_backend/app/db.pysurfsense_backend/app/utils/document_converters.pysurfsense_backend/app/indexing_pipeline/indexing_pipeline_service.pysurfsense_backend/app/agents/new_chat/system_prompt.pysurfsense_backend/app/agents/new_chat/tools/registry.pysurfsense_backend/app/templates/export_helpers.pysurfsense_backend/app/templates/report_pdf.typstsurfsense_backend/app/templates/report_html.csssurfsense_backend/app/routes/reports_routes.pysurfsense_backend/app/routes/notifications_routes.pysurfsense_backend/app/routes/search_source_connectors_routes.pysurfsense_backend/app/routes/youtube_routes.pysurfsense_backend/app/routes/documents_routes.pysurfsense_backend/app/routes/new_chat_routes.pysurfsense_backend/app/indexing_pipeline/adapters/file_upload_adapter.pysurfsense_backend/app/tasks/celery_tasks/document_tasks.pysurfsense_backend/app/users.pysurfsense_backend/app/connectors/google_gmail_connector.pysurfsense_backend/app/connectors/composio_gmail_connector.pysurfsense_web/app/(home)/page.tsxsurfsense_web/components/layout/providers/LayoutDataProvider.tsxsurfsense_web/components/layout/ui/sidebar/DocumentsSidebar.tsxsurfsense_web/components/layout/ui/right-panel/RightPanel.tsxsurfsense_web/app/dashboard/[search_space_id]/documents/(manage)/components/DocumentsTableShell.tsxsurfsense_web/app/dashboard/[search_space_id]/documents/(manage)/components/DocumentsFilters.tsxsurfsense_web/components/new-chat/document-mention-picker.tsxsurfsense_web/components/assistant-ui/inline-mention-editor.tsxsurfsense_web/components/assistant-ui/thread.tsxsurfsense_web/components/assistant-ui/assistant-message.tsxsurfsense_web/components/chat-comments/comment-panel-container/comment-panel-container.tsxsurfsense_web/components/report-panel/report-panel.tsxsurfsense_web/atoms/agent-tools/agent-tools.atoms.tssurfsense_web/atoms/chat/mentioned-documents.atom.tssurfsense_web/atoms/layout/right-panel.atom.tssurfsense_web/app/dashboard/[search_space_id]/new-chat/[[...chat_id]]/page.tsxsurfsense_web/app/dashboard/[search_space_id]/settings/page.tsxsurfsense_web/app/dashboard/[search_space_id]/user-settings/page.tsxsurfsense_web/app/dashboard/[search_space_id]/user-settings/components/ProfileContent.tsxsurfsense_web/app/dashboard/[search_space_id]/user-settings/components/ApiKeyContent.tsxsurfsense_web/components/homepage/navbar.tsxsurfsense_web/components/homepage/github-stars-badge.tsxsurfsense_web/app/layout.tsxsurfsense_web/app/sitemap.tsREADME.mdREADME.es.mdREADME.hi.mdREADME.pt-BR.mdREADME.zh-CN.mdbiome.jsonskills-lock.json.cursor/skills/shadcn/SKILL.md.cursor/skills/vercel-react-best-practices/SKILL.md.cursor/skills/web-design-guidelines/SKILL.md