Skip to content

[feat](iceberg) HDFS lazy open + iceberg delete file file_size propagation - #66773

Open
cambyzju wants to merge 3 commits into
apache:masterfrom
cambyzju:opt-master-lazyopen-260814
Open

[feat](iceberg) HDFS lazy open + iceberg delete file file_size propagation#66773
cambyzju wants to merge 3 commits into
apache:masterfrom
cambyzju:opt-master-lazyopen-260814

Conversation

@cambyzju

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:
When block cache hits, HdfsFileReader still calls hdfsOpenFile on every construction, wasting NameNode RPCs for data that's never read from HDFS.

How To Fix:

  1. HdfsFileHandle now lazily opens on first read;
  2. FE propagates delete file file_size through thrift to BE;

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@cambyzju

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17642 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9c0e9b19a3ab6b4f51efbc3b4866347c617d6e12, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17649	3143	3130	3130
q2	1887	236	154	154
q3	10465	956	510	510
q4	4673	251	204	204
q5	7668	560	373	373
q6	137	117	98	98
q7	538	504	392	392
q8	9246	901	960	901
q9	3461	2357	2388	2357
q10	6521	876	704	704
q11	440	257	242	242
q12	708	395	338	338
q13	17879	1860	1522	1522
q14	167	151	138	138
q15	q16	432	406	363	363
q17	803	817	856	817
q18	3081	2256	2223	2223
q19	1295	911	772	772
q20	685	530	475	475
q21	5750	1704	2056	1704
q22	328	275	225	225
Total cold run time: 93813 ms
Total hot run time: 17642 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3510	3460	3439	3439
q2	218	218	159	159
q3	2187	2323	2175	2175
q4	1189	1188	901	901
q5	2197	2131	2108	2108
q6	175	123	88	88
q7	1038	934	820	820
q8	1621	1440	1420	1420
q9	3086	3045	3030	3030
q10	1881	1796	1618	1618
q11	353	273	251	251
q12	461	442	348	348
q13	1807	1867	1558	1558
q14	175	184	169	169
q15	q16	393	394	367	367
q17	1049	1048	1041	1041
q18	4948	4385	4728	4385
q19	891	850	859	850
q20	978	935	828	828
q21	3883	3182	3438	3182
q22	383	359	326	326
Total cold run time: 32423 ms
Total hot run time: 29063 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 85447 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9c0e9b19a3ab6b4f51efbc3b4866347c617d6e12, data reload: false

query5	4223	416	344	344
query6	395	162	152	152
query7	4885	427	267	267
query8	292	121	107	107
query9	8678	2901	2913	2901
query10	380	271	215	215
query11	5377	1021	902	902
query12	118	71	70	70
query13	1202	469	339	339
query14	5952	2005	1885	1885
query14_1	1784	1769	1757	1757
query15	179	111	111	111
query16	918	388	357	357
query17	793	445	368	368
query18	2339	325	235	235
query19	170	146	122	122
query20	70	70	68	68
query21	208	128	103	103
query22	5564	5357	5370	5357
query23	7289	6614	6597	6597
query23_1	6549	6668	6556	6556
query24	7297	1102	778	778
query24_1	804	783	803	783
query25	442	306	276	276
query26	1267	279	177	177
query27	2695	423	277	277
query28	4665	1514	1519	1514
query29	938	457	358	358
query30	276	182	153	153
query31	975	657	620	620
query32	95	52	52	52
query33	460	216	189	189
query34	1005	812	459	459
query35	407	396	349	349
query36	566	553	532	532
query37	124	85	76	76
query38	1005	855	793	793
query39	536	547	527	527
query39_1	524	510	476	476
query40	225	122	114	114
query41	60	59	57	57
query42	79	78	77	77
query43	253	252	221	221
query44	1021	564	573	564
query45	114	104	101	101
query46	809	806	532	532
query47	1007	970	893	893
query48	328	322	230	230
query49	542	270	195	195
query50	778	332	263	263
query51	8053	8060	8017	8017
query52	72	75	60	60
query53	208	217	162	162
query54	220	177	192	177
query55	75	56	55	55
query56	230	217	257	217
query57	650	634	635	634
query58	221	200	200	200
query59	1127	1117	995	995
query60	243	227	203	203
query61	117	112	109	109
query62	374	205	180	180
query63	199	160	152	152
query64	2706	677	527	527
query65	1562	1568	1525	1525
query66	1774	300	246	246
query67	9766	9870	9588	9588
query68	2882	1183	802	802
query69	326	232	189	189
query70	648	602	595	595
query71	299	245	234	234
query72	2494	1781	1543	1543
query73	660	585	360	360
query74	1845	1218	1137	1137
query75	1235	1149	1006	1006
query76	2291	753	548	548
query77	243	273	206	206
query78	5239	4768	4441	4441
query79	1187	813	592	592
query80	1151	388	347	347
query81	469	200	170	170
query82	639	137	110	110
query83	316	254	241	241
query84	296	124	102	102
query85	856	437	374	374
query86	386	182	170	170
query87	1008	963	884	884
query88	2798	2126	2141	2126
query89	303	233	212	212
query90	1841	149	144	144
query91	154	147	124	124
query92	49	46	47	46
query93	1367	1185	789	789
query94	618	265	213	213
query95	613	365	349	349
query96	826	583	277	277
query97	1058	1064	1039	1039
query98	141	139	132	132
query99	411	345	310	310
Total cold run time: 178297 ms
Total hot run time: 85447 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9c0e9b19a3ab6b4f51efbc3b4866347c617d6e12, data reload: false

query1	0.01	0.00	0.01
query2	0.08	0.03	0.03
query3	0.25	0.11	0.11
query4	1.60	0.10	0.09
query5	0.17	0.16	0.16
query6	1.26	0.70	0.68
query7	0.03	0.01	0.00
query8	0.05	0.03	0.03
query9	0.30	0.20	0.20
query10	0.35	0.36	0.33
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.30	0.32	0.30
query14	0.49	0.45	0.46
query15	0.35	0.34	0.35
query16	0.21	0.22	0.22
query17	0.69	0.73	0.64
query18	0.20	0.17	0.17
query19	1.15	1.15	1.17
query20	0.02	0.01	0.01
query21	15.43	0.15	0.11
query22	5.08	0.05	0.04
query23	16.19	0.27	0.12
query24	2.96	0.32	0.27
query25	0.12	0.04	0.04
query26	0.80	0.17	0.12
query27	0.04	0.02	0.03
query28	3.64	0.56	0.28
query29	12.41	3.19	2.59
query30	0.26	0.11	0.12
query31	2.76	0.36	0.17
query32	3.53	0.32	0.23
query33	1.37	1.41	1.37
query34	15.39	2.17	1.75
query35	1.74	1.72	1.72
query36	0.46	0.29	0.29
query37	0.07	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.11	0.08	0.07
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.03	0.02	0.03
Total cold run time: 90.4 s
Total hot run time: 14.6 s

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.23% (2001/2558)
Line Coverage 65.30% (36508/55907)
Region Coverage 65.49% (18564/28347)
Branch Coverage 55.14% (10106/18328)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants