Respect explicitly requested H.264 High Profile in hardware encoder s…

[rajeshgk]

Summary

This pull request proposes a fix for H.264 hardware encoding in FFmpegWriter.cpp, where the encoder initialization unconditionally forces AV_PROFILE_H264_CONSTRAINED_BASELINE, overriding an explicitly requested High Profile.

The proposed change honors AV_PROFILE_H264_HIGH when explicitly selected, while retaining Constrained Baseline as the legacy default.

Changes

• Detect an explicitly requested H.264 High Profile.
• Preserve High Profile instead of overwriting it with Constrained Baseline.
• Set the encoder-private vprofile option consistently.
• Preserve the existing default behavior for other profile values.

Environment

• OpenShot 4.0.1 AppImage
• Ubuntu Linux
• Dell Precision M4800
• Intel HD Graphics 4600
• Intel VAAPI driver: i965
• Encoder: h264_vaapi
• Export: 1280 × 720, 30 FPS

Root cause

The H.264 hardware-encoding branch in src/FFmpegWriter.cpp contains:

video_codec_ctx->profile = AV_PROFILE_H264_CONSTRAINED_BASELINE;

This overrides a High Profile request made through:

w.SetOption(openshot.VIDEO_STREAM, "profile", "100")

Experimental verification

A local experimental binary modification changed the H.264 profile assignment from 578 (Constrained Baseline) to 100 (High Profile).

The resulting OpenShot VAAPI export was verified using ffprobe:

codec_name=h264
profile=High
width=1280
height=720
pix_fmt=yuv420p
level=31

This demonstrates successful H.264 High Profile output on the tested Intel hardware.

Testing status

• Original Constrained Baseline behavior reproduced
• Root cause identified in FFmpegWriter
• Experimental binary workaround verified
• Proposed C++ source change committed
• Modified C++ source compiled
• Source-level fix tested against Intel VAAPI
• Regression testing on other hardware encoders

Request for review

I would appreciate feedback from the OpenShot maintainers on the proposed profile-selection logic and hardware compatibility.

This pull request is submitted as a draft because the source-level implementation still requires compilation and regression testing.

Thank you for maintaining OpenShot and supporting open-source video editing.

[rajeshgk]

H.264 High Profile — Runtime Verification Results

The proposed source-level change has been tested successfully on Intel HD Graphics 4600 using the i965 VAAPI driver.

Test 1 — Source-built libopenshot

• Encoder: h264_vaapi
• Profile explicitly requested using SetOption(VIDEO_STREAM, "profile", "100")
• Resolution: 1280 × 720
• Frame rate: 30 fps
• Frames encoded: 150
• Verified output: H.264 High Profile
• Encoding completed successfully.

Test 2 — Experimental OpenShot 4.0.1 AppImage

• Used the previously patched AppImage library and modified Python export configuration.
• Exported 2,892 frames at 1280 × 720.
• Duration: 96.4 seconds
• Average bitrate: 913,055 bps
• Output file size: 11,002,316 bytes
• FFprobe confirmed profile=High, pix_fmt=yuv420p, and H.264 Level 3.1.
• Full FFmpeg decoding completed without reported errors.

Build status: The source library and Python SWIG binding built successfully. The full default build encountered an unrelated unresolved linking failure in an optional example target.

Scope clarification: The GUI test validates the experimental AppImage binary patch and Python export integration. It does not constitute a GUI integration test of the newly compiled source library.

Latest source commit: 8f5a3aa2.

[rajeshgk]

Final Regression Verification — PASSED

The H.264 High Profile enhancement has now passed both positive and backward-compatibility regression tests.

Environment

• Intel HD Graphics 4600
• Intel i965 VAAPI driver
• Source-built libopenshot with FFmpeg 7

Test 1 — Explicit High Profile

• Encoder: h264_vaapi
• Profile explicitly set to 100
• Resolution: 1280 × 720
• Frames encoded: 150
• FFprobe result: profile=High
• Result: PASS

Test 2 — Default Constrained Baseline

• Encoder: h264_vaapi
• No explicit profile override
• Resolution: 1280 × 720
• Frames encoded: 150
• FFprobe result: profile=Constrained Baseline
• Encoding exit status: 0
• Full FFmpeg decoding exit status: 0
• Result: PASS

Additional GUI validation

An experimental OpenShot 4.0.1 AppImage with the corresponding binary patch and Python export setting successfully produced a 96.4-second, 2,892-frame H.264 High Profile video. Full FFmpeg decoding completed without errors.

The GUI test used the experimental binary patch, not the newly source-built library.

Conclusion

The source patch correctly honors an explicitly requested H.264 High Profile while preserving the existing Constrained Baseline default.

Latest commit: 8f5a3aa2.

All reported runtime regression tests passed. The full default build's optional-example linking issue remains unresolved.