Skip to content

Explore a standalone, dependency-free R package for the Data Viewer backend #1860

Open
Open
Explore a standalone, dependency-free R package for the Data Viewer backend#1860
Assignees
eitsupi

Description

@eitsupi
eitsupi
opened on Oct 11, 2026

Motivation

The Data Viewer backend is currently implemented inside sess, which depends on external R packages such as jsonlite, processx, and later.
These dependencies can complicate installation and cause conflicts with packages managed by renv, pixi, or other environment managers.

PR #1856 improves installation isolation, but does not eliminate these dependencies.

I propose exploring a small, standalone R package for the Data Viewer backend, similar to how jgd provides a graphics device without external R package dependencies.

Unlike session communication, the enhanced Data Viewer is optional.
If its backend is unavailable, only the enhanced viewer needs to be disabled, without affecting the R session or the standard View() behavior.

The primary goal is dependency isolation and maintainability, not performance optimization.

Proposed direction

  • Publish the backend as an independent CRAN package, installed into a vscode-R-managed library rather than bundled with the extension.
  • Avoid external R package dependencies (Imports / LinkingTo), using base R and a small amount of C code where appropriate.
  • Keep the backend independent of sess, and VS Code-specific communication protocols.
  • Reuse existing R methods such as format() for displaying S3/S4 objects, including vector classes built with vctrs.
  • Preserve native comparison and sorting semantics independently of formatted display values.
  • Generate only the JSON structures required by the Data Viewer, potentially using an embedded C library such as cJSON.
  • Keep the existing AG Grid frontend.

A general-purpose serializer such as jsonlite provides much more functionality than the Data Viewer needs.
By normalizing R values into a limited set of display types before JSON encoding, we may be able to avoid both a jsonlite dependency and vendoring its implementation.

Prior art

  • jgd: A standalone R graphics device with embedded C code and no external R package dependencies.
  • vctrs / pillar: Established conventions for separating vector formatting, type information, and comparison semantics.
  • Julia VS Code Table Viewer: Converts language-specific values into a limited JSON representation for AG Grid, with lazy data retrieval.

Initial investigation

A prototype could focus on ordinary data frames and matrices, evaluating:

  1. Whether the existing metadata, paging, sorting, and filtering behavior can be extracted into a standalone package.
  2. Whether existing R formatting and comparison methods are sufficient for custom vector classes, including those based on vctrs.
  3. Whether a minimal JSON encoder can preserve existing behavior for missing values, numeric precision, dates, integer64, and complex columns.
  4. Whether the package can be installed and loaded from a managed library without modifying .libPaths().

This is an architectural investigation, not a proposal to immediately replace the current implementation.

PR #1804 (lazy Arrow/DBI viewing) should proceed independently.
Arrow IPC / nanoarrow are not required for the initial prototype.

Separating the Data Viewer also does not eliminate sess's own jsonlite dependency, which is a separate concern related to #1859.
However, an independent backend could eventually be reused through alternative transports such as arf.

Related: #1856, #1859, #1804.

Activity

  1. Fred-Wu commented on Oct 11, 2026

    @Fred-Wu
    Fred-Wu
    on Oct 11, 2026
    Contributor

    As I remember, before v3.0, the R-side Data Viewer backend was largely implemented in a single file, including operations such as data preparation, formatting, sorting, filtering, paginated row fetching, and viewer register.

    Before v3.0, I implemented on-demand data loading using httpuv in my own local build, and the current work evolved from that one.

    So I think extracting these operations into a standalone package should be feasible.

  2. eitsupi
    self-assigned this
    on Oct 13, 2026

  3. eitsupi commented on Oct 13, 2026

    @eitsupi
    eitsupi
    on Oct 13, 2026
    MemberAuthor

    I'm working on this.

    Image

  4. eitsupi commented on Oct 15, 2026

    @eitsupi
    eitsupi
    on Oct 15, 2026
    MemberAuthor

    I've made https://github.com/eitsupi/jdv public.

    @Fred-Wu Could you take a look at this?
    While DBI/dbplyr support is not (yet) available, I believe this should cover what you want to do in #1876.

    Database support is already covered if we use ADBC, but I'm not sure if that meets your needs.
    I'm still considering whether to add lazy R support.

  5. Fred-Wu commented on Oct 16, 2026

    @Fred-Wu
    Fred-Wu
    on Oct 16, 2026
    Contributor

    Thanks for working on this @eitsupi. It looks promising.

    Here are some immediate issues after trying the package

    1. Filtering for date and date/time columns doesn't seem to work, always resulting in error for a very large data.
      The current implementation may convert entire date and date/time columns into formatted strings before filtering. Filtering and sorting should use the underlying numeric values directly.
    2. If sorting or filtering is applied on a column, loading to the next page is slow. So need to check whether repeated filter is performed when loading subsequent pages.
    3. Changes made on a data.frame do not reflect on the table when clicking on Open view again.
    4. Initial table should only check column metadata using zero-row slice, deferring checks until needed
    5. Filtering should include blank and notBlank filters in addition to the NA filter.
    6. For List viewer, it should avoid repeated individual child inspection and formatting for vectors, only format on requested pages together.
    7. Are those query_limits all necessary? For example, some genomic datasets may easily surpass 4096 columns.
    8. The maximum cache_bytes limits to 32mb. For very large datasets, cached sorting or filtering results may exceed this limit and cannot be retained, or may be discarded when other cached data consumes the available space. This can cause expensive operations to be repeated when loading subsequent pages as observed in 2.

    Here is the testing data I am using.
    lazy_query_test.txt

  6. eitsupi commented on Oct 17, 2026

    @eitsupi
    eitsupi
    on Oct 17, 2026
    MemberAuthor

    @Fred-Wu Thanks again for testing jdv and for the detailed feedback!
    I have pushed fixes for most of the issues you reported, including date filtering on large data, slow paging under sort/filter, and the column limit.
    Re-running jdv_view(x) also now picks up changes to the variable automatically.
    Could you try the latest main again with your test data when you have time?

    If anything still looks slow or doesn't update, could you let me know which object and which viewer (the jdv browser viewer or the vscode-R PR) you were using?

  7. Fred-Wu commented on Oct 18, 2026

    @Fred-Wu
    Fred-Wu
    on Oct 18, 2026
    Contributor

    7. If anything still looks slow or doesn't update, could you let me know which object and which viewer (the jdv browser viewer or the vscode-R PR) you were using?

    Thanks @eitsupi. I have tried jdv with the vscode-R PR in the data viewer (but not the latest push). Some quick comments below.

    1. Normal scrolling for data frames still feels slightly unsmooth. Review suggests that fetchGridPage() in jdvViewer.ts may send repeated requests to fill the rows requested by AG Grid, potentially due to the max_message_bytes and row_capacity budget limits.
    2. Arrow numeric values appear as Invalid number in the vscode-R data viewer.
    3. For Arrow Table object, the row count is already available from its metadata through nrow(). I think, row number should be made available in this case.

  8. eitsupi commented on Oct 19, 2026

    @eitsupi
    eitsupi
    on Oct 19, 2026
    MemberAuthor

    @Fred-Wu Thanks, could you try the latest commit?

    I couldn't get any information about performance. There seem to be no problems with the jdv communication path.

  9. Fred-Wu commented on Oct 19, 2026

    @Fred-Wu
    Fred-Wu
    on Oct 19, 2026
    Contributor

    @eitsupi I have tried the latest commit.

    1. It seems that not all uni-codes are supported for data frames.

    2. The performance issue is associated with wide data. Here is the comparison between sess and jdv in the data viewer. The data contains 686K rows and 150 columns, and only Year and Month columns were shown. Others were hidden.

    sess

    sess.mp4

    jdv

    jdv.mp4

  10. eitsupi commented on Oct 19, 2026

    @eitsupi
    eitsupi
    on Oct 19, 2026
    MemberAuthor

    Thanks @Fred-Wu

    1. Unicode: I couldn't reproduce this in a UTF-8 locale. Which locale (Sys.getlocale()) and what kind of data were you using? Does the same problem happen with the sess viewer?
    2. Performance: I've pushed some improvements. The viewer now fetches only the displayed (non-hidden) columns, and receiving large responses is faster. Could you try the latest commit again? Fetching only the columns in the visible horizontal range, so that wide tables with all columns shown are faster too, is planned as a follow-up.

  11. Fred-Wu commented on Oct 19, 2026

    @Fred-Wu
    Fred-Wu
    on Oct 19, 2026
    Contributor
    1. For unicodes, sess viewer has no issue. jdv Arrow Table is correct, but not data frames. Here is my locale and unicodes tested.
    R> Sys.getlocale()
[1] "LC_COLLATE=English_Australia.1252;LC_CTYPE=English_Australia.1252;LC_MONETARY=English_Australia.1252;LC_NUMERIC=C;LC_TIME=English_Australia.1252"

unicode <- c("α", "β", "γ", "é", "è", "ç", "ñ", "ü", "中文", "汉字", "北京", "日本語", "東京", "こんにちは", "한글", "한국어", "العربية", "हिन्दी", "Ω", "π", "∞", "€", "😊"),

    Arrow Table
    Image

    data.frame
    Image

    1. As for performance, what the previous review suggests is that jdv may send repeated requests to fill the rows requested by AG Grid. The default row fetching is 500 per page. However, due to max_message_bytes and row_capacity budget limits, it may only get 250 rows first, then another 250 rows to fill the requested 500 rows. Here is another demo on a 30M x 11 data.

    sess

    sess.mp4

    jdv

    jdv.mp4

  12. eitsupi commented on Oct 20, 2026

    @eitsupi
    eitsupi
    on Oct 20, 2026
    MemberAuthor

    @Fred-Wu Thanks for the report!
    I fixed the locale issue and the polling problem on Windows.
    Could you try again?

  13. Fred-Wu commented on Oct 21, 2026

    @Fred-Wu
    Fred-Wu
    on Oct 21, 2026
    Contributor

    @eitsupi Thanks for fixing those quickly. Unicode issue has been fixed, and scrolling is now smooth for data frames after the polling change.

    1. One suggestion for Arrow Table, the Arrow stream label sits beside the buttons, maybe put it before row count?
    2. A further issue is that date and date/time formats are lost in List Viewer.
    3. Formatting from an arbitrary class is not preserved in List Viewer. Formatting should be applied to any class having a generic format method.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions