Skip to content

Create a PET Accesors for Pangeo CMIP6 cloud hosted data #276

Open
0 of 3 issues completed
Open
Create a PET Accesors for Pangeo CMIP6 cloud hosted data#276
0 of 3 issues completed
Assignees
ahmedmohiduet
Labels
iccs-hacktoberfest

Description

@stevehadd
stevehadd
opened on Sep 25, 2026

The CMIP6 dataset is very valuable. The Pangeo community hosted an Analsysis-ready Cloud-optised or ARCO version of this data. This issue is create an ArchiveIndex derived class to handle accessing and loading data from the cloud

Tasks

  • Create a PET class to handle accessing and loading the data.
    • use the ArchiveIndex classes in packages/third_party/site_archive_aws for inspiration.
  • Create a notebook to demonstrate loading and visualising various data
  • Create a notebook to demonstrate training a simple model (e.g. autoencoder) using CMIP6 data and PET training wrapper
    • Look at training wrapper tutorial for guidance on training in PET
    • It is recommended to include a Cache step in your pipeline to cache the normalised data locally to reduce loading times and get data onto the GPU faster, reducing GPU downtime between batches.

Further Reading

Activity

  1. stevehadd
    added
    iccs-hacktoberfest
    on Sep 25, 2026

  2. ahmedmohiduet commented on Oct 2, 2026

    @ahmedmohiduet
    ahmedmohiduet
    on Oct 2, 2026

    Hi @stevehadd, I'm taking part in ICCS Hacktoberfest and would like to work on this.

    Since there are three distinct tasks, would it be OK to split them into separate PRs?

    1. ArchiveIndex-derived class for the Pangeo CMIP6 ARCO data, modelled on the classes in packages/third_party/site_archive_aws, with tests using small synthetic data so CI doesn't need network access
    2. Notebook demonstrating loading and visualising the data
    3. Notebook training a simple autoencoder with the PET training wrapper, including a Cache step

    A couple of questions before I start:

    • Do you have a preference between using intake-esm with the Pangeo catalog, or reading the catalog CSV directly to avoid an extra dependency?
    • For the training notebook, is a small, coarse subset (e.g. one variable, one model, a few years) fine, so it runs in reasonable time without a large GPU?
  3. stevehadd
    added sub-issues on Oct 2, 2026

  4. stevehadd commented on Oct 2, 2026

    @stevehadd
    stevehadd
    on Oct 2, 2026
    CollaboratorAuthor

    Hi @ahmedmohiduet really glad you'd like to take part in hacktoberfest and contribute to PyEarthTools.

    I think it would be good to split it up into three PRs. In fact I should really have created 3 sub-issues for each of the pieces of work, so I have now done that and assigned the subissues to you. Then each PR can close a sub issue.

    On the technical questions

    • I have no particular preference, there are pros and cons to both, happy to go with whichever solution you think is best. Perhaps you can give some explanation in your PR description as to why you chose one over the other
    • For training, please do you a subset of the data to demonstrate, since as you say we don't this to take too long or be too demanding on resources. It would be good for the code to be easy to change so that you can select a much larger dataset for training quite easily, without having to change the pipeline, just change some parameter like a list of variable and a tuple of start and end datetime objects.

    Just tagging @tennlee who is principal maintainer to see if he has any additional thoughts on the implementation, do make a start in the meanwhile, we can always adjust implementation specifics later.

  5. stevehadd
    assigned
    ahmedmohiduet
    on Oct 2, 2026

  6. ahmedmohiduet commented on Oct 3, 2026

    @ahmedmohiduet
    ahmedmohiduet
    on Oct 3, 2026

    Thanks @stevehadd, that's really helpful, and thanks for setting up the sub-issues. I'll start with #284 (the accessor class) and keep the variables and time range as parameters throughout, so later notebooks can scale up easily. I'll explain the catalog approach in the PR description.

  7. ahmedmohiduet commented on Oct 3, 2026

    @ahmedmohiduet
    ahmedmohiduet
    on Oct 3, 2026

    @stevehadd small thing: it looks like only #276 got assigned to me, and the sub-issues #284, #285 and #286 still show no assignee. Could you assign those to me as well when you get a chance? Thanks!

  8. stevehadd commented on Oct 3, 2026

    @stevehadd
    stevehadd
    on Oct 3, 2026
    CollaboratorAuthor

    @stevehadd small thing: it looks like only #276 got assigned to me, and the sub-issues #284, #285 and #286 still show no assignee. Could you assign those to me as well when you get a chance? Thanks!

    I seem to be having some trouble asigning these issues. I have assigned #284 for now, and will try to figure out why I can't assign the others and sort those out too.

  9. stevehadd commented on Oct 3, 2026

    @stevehadd
    stevehadd
    on Oct 3, 2026
    CollaboratorAuthor

    @ahmedmohiduet I've figured it out. Could you please comment on each of the issues, then I am able to assign them to you.

  10. ahmedmohiduet commented on Oct 3, 2026

    @ahmedmohiduet
    ahmedmohiduet
    on Oct 3, 2026

    Thanks @stevehadd, I've commented on #285 and #286.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

iccs-hacktoberfest

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions