Skip to content

[fix](fe) fix Azure resource persistence#65052

Open
gavinchou wants to merge 1 commit into
apache:masterfrom
gavinchou:gavin-fix-azure-resource-gson
Open

[fix](fe) fix Azure resource persistence#65052
gavinchou wants to merge 1 commit into
apache:masterfrom
gavinchou:gavin-fix-azure-resource-gson

Conversation

@gavinchou

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: DORIS-26664

Related PR: None

Problem Summary:

AzureResource was not registered in the Resource runtime Gson adapter. As a result, persisted Azure resources could be serialized without the clazz field, and FE replay from edit log or image failed when deserializing them as Resource.class.

This PR registers AzureResource in GsonUtils and makes Resource.read compatible with legacy resource JSON that has type but no clazz, deriving the subtype label from ResourceType.

Release note

None

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

Added unit coverage in ResourcePersistTest. Not run locally per request.

  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@gavinchou

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29622 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f1c19744eb87ff29c5ab2984e4a5cfa4fd71390, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17728	4092	4027	4027
q2	2086	310	199	199
q3	10283	1425	840	840
q4	4680	470	340	340
q5	7515	839	579	579
q6	180	173	143	143
q7	759	835	620	620
q8	9824	1508	1504	1504
q9	5924	4356	4379	4356
q10	6831	1785	1516	1516
q11	519	343	317	317
q12	734	561	435	435
q13	18058	3393	2778	2778
q14	275	261	240	240
q15	q16	793	768	719	719
q17	1004	1039	1042	1039
q18	6740	5787	5572	5572
q19	1594	1265	1101	1101
q20	828	646	522	522
q21	5725	2708	2475	2475
q22	440	358	300	300
Total cold run time: 102520 ms
Total hot run time: 29622 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4337	4429	4414	4414
q2	286	312	216	216
q3	4546	4932	4427	4427
q4	2058	2176	1371	1371
q5	4444	4280	4321	4280
q6	232	174	128	128
q7	1737	2121	1601	1601
q8	2495	2182	2054	2054
q9	7843	7900	7767	7767
q10	4771	4750	4280	4280
q11	593	420	374	374
q12	755	749	540	540
q13	3429	3587	2994	2994
q14	291	322	273	273
q15	q16	761	768	625	625
q17	1358	1326	1299	1299
q18	7906	7280	7126	7126
q19	1148	1106	1062	1062
q20	2236	2201	1917	1917
q21	5266	4587	4434	4434
q22	531	475	404	404
Total cold run time: 57023 ms
Total hot run time: 51586 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 174374 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f1c19744eb87ff29c5ab2984e4a5cfa4fd71390, data reload: false

query5	4313	628	481	481
query6	457	227	215	215
query7	4838	570	345	345
query8	380	187	177	177
query9	8761	4105	4094	4094
query10	458	375	302	302
query11	5955	2342	2175	2175
query12	172	102	100	100
query13	1419	584	460	460
query14	6283	5307	4971	4971
query14_1	4360	4352	4305	4305
query15	219	212	182	182
query16	1059	472	448	448
query17	1147	776	588	588
query18	2691	481	344	344
query19	254	190	156	156
query20	118	111	114	111
query21	245	160	137	137
query22	13577	13672	13425	13425
query23	17266	16570	16093	16093
query23_1	16227	16229	16299	16229
query24	7582	1834	1321	1321
query24_1	1330	1303	1298	1298
query25	586	477	423	423
query26	1361	371	213	213
query27	2584	574	362	362
query28	4457	2072	2049	2049
query29	1115	646	535	535
query30	361	269	224	224
query31	1110	1093	992	992
query32	106	62	61	61
query33	539	314	242	242
query34	1207	1102	638	638
query35	771	789	675	675
query36	1406	1417	1212	1212
query37	169	105	93	93
query38	1913	1673	1664	1664
query39	917	926	921	921
query39_1	877	883	894	883
query40	267	163	148	148
query41	65	63	70	63
query42	96	90	91	90
query43	325	320	285	285
query44	1450	808	782	782
query45	210	202	191	191
query46	1080	1229	749	749
query47	2333	2365	2175	2175
query48	395	410	288	288
query49	623	420	313	313
query50	1031	424	349	349
query51	4419	4386	4402	4386
query52	86	87	74	74
query53	268	284	209	209
query54	273	247	210	210
query55	75	71	68	68
query56	283	294	288	288
query57	1425	1414	1305	1305
query58	301	277	249	249
query59	1565	1646	1486	1486
query60	310	263	246	246
query61	182	149	148	148
query62	688	651	590	590
query63	255	212	216	212
query64	2542	770	618	618
query65	4870	4800	4794	4794
query66	1816	502	385	385
query67	29793	29680	29529	29529
query68	3139	1696	947	947
query69	414	298	262	262
query70	1038	954	973	954
query71	356	314	320	314
query72	2799	2662	2283	2283
query73	889	808	441	441
query74	5117	4985	4829	4829
query75	2611	2606	2212	2212
query76	2356	1181	793	793
query77	341	386	291	291
query78	12293	12394	11960	11960
query79	1196	1182	730	730
query80	603	549	447	447
query81	461	324	280	280
query82	231	156	120	120
query83	311	320	293	293
query84	283	160	127	127
query85	902	638	507	507
query86	320	297	290	290
query87	1844	1850	1775	1775
query88	3707	2811	2821	2811
query89	431	407	369	369
query90	2175	196	197	196
query91	202	187	161	161
query92	62	59	57	57
query93	1598	1581	1002	1002
query94	560	360	311	311
query95	798	501	493	493
query96	1100	776	350	350
query97	2688	2673	2559	2559
query98	226	206	202	202
query99	1165	1156	1023	1023
Total cold run time: 258274 ms
Total hot run time: 174374 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f1c19744eb87ff29c5ab2984e4a5cfa4fd71390, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.04
query3	0.26	0.13	0.13
query4	1.60	0.14	0.14
query5	0.25	0.24	0.21
query6	1.25	1.05	1.10
query7	0.04	0.01	0.01
query8	0.06	0.03	0.03
query9	0.38	0.31	0.33
query10	0.56	0.57	0.55
query11	0.20	0.14	0.14
query12	0.18	0.14	0.14
query13	0.47	0.48	0.47
query14	1.04	1.00	1.02
query15	0.61	0.59	0.60
query16	0.30	0.31	0.32
query17	1.13	1.15	1.14
query18	0.22	0.21	0.22
query19	2.10	2.00	1.94
query20	0.01	0.01	0.02
query21	15.45	0.22	0.13
query22	4.80	0.06	0.05
query23	16.16	0.30	0.12
query24	2.97	0.43	0.32
query25	0.13	0.05	0.04
query26	0.72	0.20	0.16
query27	0.05	0.05	0.04
query28	3.50	1.00	0.54
query29	12.52	4.35	3.53
query30	0.27	0.16	0.15
query31	2.78	0.60	0.31
query32	3.23	0.60	0.49
query33	3.16	3.30	3.23
query34	15.64	4.22	3.54
query35	3.54	3.54	3.55
query36	0.56	0.44	0.43
query37	0.09	0.06	0.07
query38	0.05	0.05	0.03
query39	0.04	0.03	0.02
query40	0.17	0.17	0.15
query41	0.08	0.04	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 96.75 s
Total hot run time: 25.39 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 17.86% (5/28) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants