[flyte deck] Streaming Decks by Future-Outlier · Pull Request #2779 · flyteorg/flytekit

Future-Outlier · 2024-10-01T13:06:39Z

Tracking issue

Why are the changes needed?

we want to support streaming decks.

What changes were proposed in this pull request?

add a function publish for the Flyte Deck.

How was this patch tested?

from flytekit import ImageSpec, task, workflow
from flytekit.deck import Deck

flytekit_hash = "6b55930d0a77efc3594ebaac056f2c75024e61b5"
flytekit = f"git+https://github.com/flyteorg/flytekit.git@{flytekit_hash}"

# Define custom image for the task
custom_image = ImageSpec(packages=[flytekit],
                            apt_packages=["git"],
                            registry="localhost:30000",
                            env={"FLYTE_SDK_LOGGING_LEVEL": 10},
                         )

@task(enable_deck=False, container_image=custom_image)
def t_no_deck():
    # Deck.publish()
    print("No Deck")

@task(enable_deck=True, container_image=custom_image)
def t_deck():
    import time
    """
    1st deck only show timeline deck
    2nd will show
    """
    for i in range(3):
        Deck.publish()
        time.sleep(1)

@task(enable_deck=True, container_image=custom_image)
def t_fail_deck():
    import time

    for i in range(3):
        Deck.publish()
        time.sleep(3)
    time.sleep(10)
    raise ValueError("Failed Deck")

@workflow
def wf():
    t_no_deck()
    t_deck()
    t_fail_deck()

if __name__ == "__main__":
    from flytekit.clis.sdk_in_container import pyflyte
    from click.testing import CliRunner
    import os

    runner = CliRunner()
    path = os.path.realpath(__file__)

    result = runner.invoke(pyflyte.main,
                           ["run", path, "t_no_deck"])
    print("Local Execution: ", result.output)

    result = runner.invoke(pyflyte.main,
                           ["run", "--remote", path,"wf"])
    print("Remote Execution: ", result.output)

Setup process

single binary.

flyte: flyteorg/flyte#6053
flytekit: #2779
flyteconsole: flyteorg/flyteconsole#890

Screenshots

flytekit: this branch

NEW FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

OSS-STREAMING-DECK-small.mov

OLD FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

OSS-STREAMING-DECK-OLD-FLYTEKIT-small.mov

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

Docs link

Summary by Bito

Implements comprehensive deck functionality improvements including Deck.publish() method and generates_deck flag handling, with enhanced error messaging. Added task name context to warning messages when enable_deck is False for clearer feedback. Refactors Protocol Buffer BoolValue wrapper from base task to models layer, improving error handling and attribute validation. Changes simplify task definition interface while maintaining proper protobuf serialization.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 3

Signed-off-by: Future-Outlier <eric901201@gmail.com>

codecov · 2024-10-01T13:49:45Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.50%. Comparing base (4cee7cd) to head (c0fd23a).
Report is 4 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2779       +/-   ##
===========================================
+ Coverage   77.90%   90.50%   +12.60%     
===========================================
  Files         217      108      -109     
  Lines       22178     4760    -17418     
  Branches     2768        0     -2768     
===========================================
- Hits        17277     4308    -12969     
+ Misses       4115      452     -3663     
+ Partials      786        0      -786

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Future-Outlier <eric901201@gmail.com>

kumare3 · 2024-10-09T04:14:45Z

I would love to see an example of tqdm

kumare3 · 2024-10-09T04:34:48Z

replaces #1704

Future-Outlier · 2024-10-09T04:38:18Z

I would love to see an example of tqdm

will try it, just let me ship the MSGPACK IDL first.

EngHabu · 2024-10-09T15:57:25Z

+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">


This will make it automatically refresh the page.. probably need that on the empty deck but not sure how to do the realtime behavior once it has content (you don't want to refresh while the user is browsing around or clicking through various tabs)

Suggested change

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<meta name="viewport" content="width=device-width, initial-scale=1.0">

<meta http-equiv="refresh" content="5" >

COOL, will try it, thank you

Signed-off-by: Future-Outlier <eric901201@gmail.com>

eapolinario

This is pretty cool. It unlocks so many use cases!

I left just a couple of comments. We're very close.

eapolinario · 2025-02-04T03:28:10Z

-    .. warning::
-
-        This feature is in beta.
-


eapolinario · 2025-02-04T03:43:52Z

+        task_name = params.task_id.name
+
+        if not params.has_attr(ENABLE_DECK) or not params.enable_deck:
+            logger.warning("Deck is disabled for this task, please don't call Deck.publish()")


let's make this error message less charged. How about we say "Call to Deck.publish() will not generate flyte decks as enable_deck=False"?

eapolinario · 2025-02-04T03:46:31Z

-        if DeckField.TIMELINE.value in self.deck_fields and ctx.user_space_params is not None:
-            ctx.user_space_params.decks.append(ctx.user_space_params.timeline_deck)
+        if self.enable_deck and ctx.user_space_params is not None:
+            ctx.user_space_params.builder().add_attr(ENABLE_DECK, True)


why do we need this?

because when disable deck, we shouldn't append deck.

eapolinario · 2025-02-04T04:13:56Z

there's something going on with the serialization tests (e.g.: https://github.com/flyteorg/flytekit/actions/runs/13126446467/job/36627463041?pr=2779). Can you take a look?

Future-Outlier · 2025-02-04T04:59:30Z

there's something going on with the serialization tests (e.g.: https://github.com/flyteorg/flytekit/actions/runs/13126446467/job/36627463041?pr=2779). Can you take a look?

yes no problem

Signed-off-by: Future-Outlier <eric901201@gmail.com>

flyte-bot · 2025-02-04T06:10:08Z

Code Review Agent Run #1c8466

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: ea8b6e0..a642c9d
- flytekit/deck/deck.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

Signed-off-by: Future-Outlier <eric901201@gmail.com>

flyte-bot · 2025-02-04T10:52:39Z

Code Review Agent Run #4926be

Actionable Suggestions - 0

Review Details

Files reviewed - 2 · Commit Range: a642c9d..e9aef35
- flytekit/tools/translator.py
- tests/flytekit/unit/deck/test_deck.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

Signed-off-by: Future-Outlier <eric901201@gmail.com>

flyte-bot · 2025-02-04T16:54:24Z

Code Review Agent Run #bb9139

Actionable Suggestions - 4

tests/flytekit/unit/deck/test_deck.py - 2
- Consider consolidating duplicate context assertions · Line 270-276
- Consider consolidating repeated assertions · Line 282-301
flytekit/core/context_manager.py - 1
- Missing return type hint for builder · Line 153-154
flytekit/deck/deck.py - 1
- Consider attribute existence check for params · Line 90-90

Review Details

Files reviewed - 8 · Commit Range: e9aef35..b649f8f
- flytekit/core/base_task.py
- flytekit/core/constants.py
- flytekit/core/context_manager.py
- flytekit/core/type_engine.py
- flytekit/deck/deck.py
- flytekit/remote/remote.py
- tests/flytekit/unit/core/test_unions.py
- tests/flytekit/unit/deck/test_deck.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

…rotobufs Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

flyte-bot · 2025-02-05T00:24:24Z

Code Review Agent Run #c1727c

Actionable Suggestions - 1

flytekit/core/base_task.py - 1
- Consider using BoolValue for generates_deck · Line 180-180

Review Details

Files reviewed - 3 · Commit Range: b649f8f..0565282
- flytekit/core/base_task.py
- flytekit/models/task.py
- tests/flytekit/unit/test_translator.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

Signed-off-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: pingsutw <pingsutw@apache.org>

flyte-bot · 2025-02-05T06:28:57Z

Code Review Agent Run #a1d78e

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: 0565282..c0fd23a
- flytekit/deck/deck.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

Future-Outlier added 2 commits October 1, 2024 21:05

[Flyte Decl] Streaming Decks

01182b4

Signed-off-by: Future-Outlier <eric901201@gmail.com>

print

9616fc3

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Future-Outlier added 4 commits October 1, 2024 23:17

sleep more

98ae7c7

Signed-off-by: Future-Outlier <eric901201@gmail.com>

add dummy deck

4e92bb0

Signed-off-by: Future-Outlier <eric901201@gmail.com>

nit

4df18b5

Signed-off-by: Future-Outlier <eric901201@gmail.com>

dummy deck

ebb4d4e

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Future-Outlier changed the title ~~[Flyte Deck] Streaming Decks~~ [wip][Flyte Deck] Streaming Decks Oct 2, 2024

Future-Outlier added 12 commits October 2, 2024 14:12

update

99522d9

Signed-off-by: Future-Outlier <eric901201@gmail.com>

nit

c19d67d

Signed-off-by: Future-Outlier <eric901201@gmail.com>

test

67cd829

Signed-off-by: Future-Outlier <eric901201@gmail.com>

return html

06da3df

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Change Deck

6d99d69

Signed-off-by: Future-Outlier <eric901201@gmail.com>

fix

b805cd7

Signed-off-by: Future-Outlier <eric901201@gmail.com>

fix recursion error

4c97758

Signed-off-by: Future-Outlier <eric901201@gmail.com>

remove redundant code

7b3574a

Signed-off-by: Future-Outlier <eric901201@gmail.com>

add dummy deck to deck init

9b60564

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Better Dummy Deck Logic

18c994f

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Merge branch 'master' into flytekit-streaming-deck

39f39d1

Deck Publish

aabcbbb

Signed-off-by: Future-Outlier <eric901201@gmail.com>

EngHabu reviewed Oct 9, 2024

View reviewed changes

Future-Outlier added 4 commits November 25, 2024 21:48

Merge branch 'master' into flytekit-streaming-deck

9ca43f3

litn

ed56352

Signed-off-by: Future-Outlier <eric901201@gmail.com>

remove dummy deck

b559fc9

Signed-off-by: Future-Outlier <eric901201@gmail.com>

nit

fc5578f

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Future-Outlier changed the title ~~[wip][Flyte Deck] Streaming Decks~~ [flytepropeller][flyteadmin] Streaming Decks Nov 27, 2024

Future-Outlier changed the title ~~[flytepropeller][flyteadmin] Streaming Decks~~ [flyte deck] Streaming Decks Nov 27, 2024

flyte-bot reviewed Feb 4, 2025

View reviewed changes

Comment thread flytekit/core/base_task.py Outdated

eapolinario reviewed Feb 4, 2025

View reviewed changes

update

a642c9d

Signed-off-by: Future-Outlier <eric901201@gmail.com>

update

e9aef35

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Future-Outlier added 4 commits February 4, 2025 23:47

deck

04775d7

Signed-off-by: Future-Outlier <eric901201@gmail.com>

update

34a4146

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Merge branch 'master' into flytekit-streaming-deck

f30382b

add executiuon parameter test

b649f8f

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Future-Outlier commented Feb 4, 2025

View reviewed changes

Comment thread tests/flytekit/unit/deck/test_deck.py

eapolinario mentioned this pull request Feb 4, 2025

Real time deck support #1704

Closed

8 tasks

flyte-bot reviewed Feb 4, 2025

View reviewed changes

Comment thread tests/flytekit/unit/deck/test_deck.py

flyte-bot reviewed Feb 4, 2025

View reviewed changes

Comment thread tests/flytekit/unit/deck/test_deck.py

flyte-bot reviewed Feb 4, 2025

View reviewed changes

Comment thread flytekit/core/context_manager.py

flyte-bot reviewed Feb 4, 2025

View reviewed changes

Comment thread flytekit/deck/deck.py

Only wrap generates_deck in a BoolValue when moving in and out of p…

0565282

…rotobufs Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

flyte-bot reviewed Feb 5, 2025

View reviewed changes

Comment thread flytekit/core/base_task.py

pingsutw reviewed Feb 5, 2025

View reviewed changes

Comment thread flytekit/deck/deck.py Outdated

Streaming Deck

c0fd23a

Signed-off-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: pingsutw <pingsutw@apache.org>

pingsutw approved these changes Feb 5, 2025

View reviewed changes

Future-Outlier merged commit 2ef875c into master Feb 5, 2025

	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<meta http-equiv="refresh" content="5" >

Conversation

Future-Outlier commented Oct 1, 2024 • edited by flyte-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

NEW FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

OLD FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

Check all the applicable boxes

Related PRs

Docs link

Summary by Bito

Uh oh!

codecov Bot commented Oct 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kumare3 commented Oct 9, 2024

Uh oh!

kumare3 commented Oct 9, 2024

Uh oh!

Future-Outlier commented Oct 9, 2024

Uh oh!

EngHabu Oct 9, 2024

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Oct 9, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eapolinario left a comment

Choose a reason for hiding this comment

Uh oh!

eapolinario Feb 4, 2025

Choose a reason for hiding this comment

Uh oh!

eapolinario Feb 4, 2025

Choose a reason for hiding this comment

Uh oh!

eapolinario Feb 4, 2025

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Feb 4, 2025

Choose a reason for hiding this comment

Uh oh!

eapolinario commented Feb 4, 2025

Uh oh!

Future-Outlier commented Feb 4, 2025

Uh oh!

flyte-bot commented Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #1c8466

Uh oh!

flyte-bot commented Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #4926be

Uh oh!

Uh oh!

flyte-bot commented Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #bb9139

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

flyte-bot commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #c1727c

Uh oh!

Uh oh!

Uh oh!

flyte-bot commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #a1d78e

Uh oh!

Reviewers

Future-Outlier commented Oct 1, 2024 •

edited by flyte-bot

Loading

codecov Bot commented Oct 1, 2024 •

edited

Loading

flyte-bot commented Feb 4, 2025 •

edited

Loading

flyte-bot commented Feb 4, 2025 •

edited

Loading

flyte-bot commented Feb 4, 2025 •

edited

Loading

flyte-bot commented Feb 5, 2025 •

edited

Loading

flyte-bot commented Feb 5, 2025 •

edited

Loading