Repository navigation
Conversation
|
@blueorangutan package |
|
@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 18352 |
17b9459 to
bf05e6c
Compare
|
@blueorangutan package |
|
@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #13478 +/- ##
============================================
+ Coverage 20.78% 20.85% +0.07%
- Complexity 21941 22190 +249
============================================
Files 6496 6522 +26
Lines 591822 593568 +1746
Branches 72615 72961 +346
============================================
+ Hits 123017 123815 +798
- Misses 455450 456244 +794
- Partials 13355 13509 +154
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18353 |
|
@blueorangutan package |
|
@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18355 |
|
@blueorangutan package |
|
@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18371 |
|
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch. |
DaanHoogland
moved this from Backlog
to conflict/waiting for author
in CloudStack Testing
505b120 to
d1a357c
Compare
|
@blueorangutan package |
|
@blueorangutan package |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19312 |
|
[SF] Trillian Build Failed (tid-17031) |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19857 |
|
@blueorangutan test |
|
@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-17435)
|
There was a problem hiding this comment.
LGTM Tested manually by creating rules for each resource type
| Test Case Execution | Result |
|---|---|
| Upgrade a 4.23 DB to the PR build | Pass |
| Check the APIs for each role | Pass |
| Create rules for each resource type, single and all-resources, account and project | Pass |
| List rules with filters and paging | Pass |
| Update fields and webhook links | Pass |
| Delete a rule | Pass |
| Reuse names | Pass |
| Create, update and delete a rule, then check events | Pass |
| Create with missing, blank or invalid values (type, metric, condition, severity) | Pass |
| Thresholds out of range (negative, above 100 for %, non-numeric, boundaries) | Pass |
| Unknown or wrong resource and webhook IDs, system VM/VR IDs | Pass |
| Update or delete a missing rule, invalid state, overflow values | Pass |
| Load a VM (CPU, memory, disk, network) | Pass |
| Load a host (CPU, memory, load, network) | Pass |
| Fill a volume and a storage pool | Pass |
| Test each condition at, above and below the threshold | Pass |
| Change the threshold of an active rule | Pass |
| Stop, start, migrate, destroy and recover a VM | Pass |
| Detach, migrate, resize and snapshot a volume | Pass |
| Put a host or pool in maintenance, or disconnect a host | Pass |
| Expunge the VM, delete the volume or host, assign the VM to another account | Pass |
| Delete the owner account or domain, or a linked webhook | Pass |
| Keep the condition true across several runs | Pass |
| Disable and re-enable a rule | Pass |
| Set the retention to N days | Pass |
| Create all-resources VM rules as user, domain admin and root admin | Pass |
| Create all-resources volume, host and pool rules | Pass |
| Add the opt-out tag to a VM, volume and host | Pass |
| Fire an alert with one or two linked webhooks | Pass |
| Disabled webhook | Pass |
| Fire an alert with email=true, SMTP down, no recipients | Pass |
| Fire an alert with and without RabbitMQ | Pass |
| Fire an alert with and without RabbitMQ | Pass |
| Run rules on a 2-MS cluster | Pass |
| Stop and restart the evaluating MS | Pass |
| User and domain admin create Host/Pool rules or set email=true | Pass |
| User targets or manages another account's resources, rules or alerts | Pass |
| Domain admin manages rules in and outside the own domain tree | Pass |
| Set the rule limit to N, -1 and 0 (global, account, project) | Pass |
| Filter by rule, resource type, resource, severity, date and keyword | Pass |
| Invalid filters (unknown rule, resourceid without type) | Pass |
| Open the rules list as admin and as user | Pass |
| Create and edit rules in the UI | Pass |
| Enable, disable and delete (single and bulk) | Pass |
| View alert history on rule and resource pages | Pass |
github-project-automation
Bot
moved this from conflict/waiting
to Done
in CloudStack Testing
There was a problem hiding this comment.
🟡 Changes recommended
ResourceAdmin list requests can expose other domains’ and root administrators’ alert rules because the required domain-tree authorization scope is not applied.
9 open issues
What changed in this PR
Adds threshold-based resource alerts for VMs, volumes, hosts and storage pools, with rule-management APIs, periodic evaluation, notification delivery and monitoring UI workflows.
Changes:
- Introduces persistent rules, scoped evaluation, cooldowns and alert history.
- Adds webhook, email and event-bus notifications, including webhook delivery history and resend support.
- Adds monitoring pages, resource-level Alerts tabs, unit tests and Marvin smoke tests.
This pull request requires changes. Unresolved findings include critical cross-domain rule visibility; incorrect rate calculations and stale metrics; cooldown/history interactions; deletion and naming races; evaluation query scaling; missing failed webhook delivery records; smoke-test configuration; project-history scoping; and resource-selection pagination. There is also an API metric-name documentation nit.
| File | Description |
|---|---|
ui/src/views/resourcealert/resourceAlertOptions.js |
Defines metrics, conditions and display labels. |
ui/src/views/resourcealert/EditResourceAlertRule.vue |
Adds the rule-editing form. |
ui/src/views/resourcealert/CreateResourceAlertRule.vue |
Adds rule creation and resource selection. |
ui/src/views/compute/InstanceTab.vue |
Adds the VM Alerts tab. |
ui/src/views/AutogenView.vue |
Enables project-scoped alert views. |
ui/src/config/section/storage.js |
Adds volume Alerts tabs. |
ui/src/config/section/monitoring.js |
Introduces the Monitoring navigation section. |
ui/src/config/section/infra/resourceAlertRules.js |
Configures rule lists, details and actions. |
ui/src/config/section/infra/primaryStorages.js |
Adds storage-pool Alerts tabs. |
ui/src/config/section/infra/hosts.js |
Adds host Alerts tabs. |
ui/src/config/router.js |
Registers Monitoring routes. |
ui/src/components/view/SearchView.vue |
Adds alert-rule search controls. |
ui/src/components/view/SearchFilter.vue |
Formats alert filters and resource names. |
ui/src/components/view/ResourceAlertsTab.vue |
Displays paginated alert history and resource-specific creation. |
ui/src/components/view/ListView.vue |
Enables rule navigation and bulk actions. |
ui/src/components/view/DetailsTab.vue |
Formats rule details and webhook names. |
ui/public/locales/en.json |
Adds English monitoring and alert-interface strings. |
tools/marvin/marvin/lib/base.py |
Adds Marvin rule-management and history helpers. |
test/integration/smoke/test_resource_alerts.py |
Tests lifecycle, permissions, delivery and cleanup. |
server/src/test/java/com/cloud/server/StatsCollectorTest.java |
Tests recent, stale and absent VM samples. |
server/src/main/resources/META-INF/cloudstack/core/spring-server-core-managers-context.xml |
Registers alert service and evaluator beans. |
server/src/main/java/org/apache/cloudstack/webhook/WebhookHelper.java |
Extends webhook lookup and direct-delivery integration. |
server/src/main/java/com/cloud/server/StatsCollector.java |
Exposes recent persisted VM statistics. |
server/src/main/java/com/cloud/api/query/vo/ResourceAlertRuleJoinVO.java |
Maps rules with ownership metadata. |
server/src/main/java/com/cloud/api/query/dao/ResourceAlertRuleJoinDaoImpl.java |
Implements rule-view UUID lookup. |
server/src/main/java/com/cloud/api/query/dao/ResourceAlertRuleJoinDao.java |
Declares rule-view query access. |
server/src/main/java/com/cloud/alert/ResourceAlertServiceImpl.java |
Implements rule APIs, validation and visibility. |
server/src/main/java/com/cloud/alert/ResourceAlertManagerImpl.java |
Evaluates rules and dispatches notifications. |
plugins/pom.xml |
Whitespace-only change. |
plugins/event-bus/webhook/src/test/java/org/apache/cloudstack/mom/webhook/WebhookServiceImplTest.java |
Tests direct delivery, filtering and event-independent resend. |
plugins/event-bus/webhook/src/main/java/org/apache/cloudstack/mom/webhook/WebhookServiceImpl.java |
Adds direct alert delivery, recording and resend support. |
plugins/event-bus/webhook/src/main/java/org/apache/cloudstack/mom/webhook/WebhookDeliveryThread.java |
Carries event types through delivery callbacks. |
plugins/event-bus/webhook/src/main/java/org/apache/cloudstack/mom/webhook/WebhookDelivery.java |
Supports deliveries without stored events. |
plugins/event-bus/webhook/src/main/java/org/apache/cloudstack/mom/webhook/vo/WebhookDeliveryVO.java |
Persists nullable event IDs and explicit event types. |
plugins/event-bus/webhook/src/main/java/org/apache/cloudstack/mom/webhook/vo/WebhookDeliveryJoinVO.java |
Maps nullable delivery event IDs. |
engine/schema/src/test/java/com/cloud/storage/dao/VolumeDaoImplTest.java |
Tests scoped user-volume selection. |
engine/schema/src/main/resources/META-INF/db/views/cloud.webhook_delivery_view.sql |
Includes deliveries without stored events. |
engine/schema/src/main/resources/META-INF/db/views/cloud.resource_alert_rule_view.sql |
Joins rules with account, domain and project metadata. |
engine/schema/src/main/resources/META-INF/db/schema-42300to2400.sql |
Creates alert tables and updates webhook deliveries. |
engine/schema/src/main/resources/META-INF/cloudstack/core/spring-engine-schema-core-daos-context.xml |
Registers alert persistence and query DAOs. |
engine/schema/src/main/java/com/cloud/vm/dao/VmStatsDaoImpl.java |
Retrieves the newest VM-stat sample. |
engine/schema/src/main/java/com/cloud/vm/dao/VmStatsDao.java |
Declares latest-sample lookup. |
engine/schema/src/main/java/com/cloud/vm/dao/UserVmDaoImpl.java |
Selects VM IDs by scope and state. |
engine/schema/src/main/java/com/cloud/vm/dao/UserVmDao.java |
Declares scoped VM-ID selection. |
engine/schema/src/main/java/com/cloud/storage/dao/VolumeDaoImpl.java |
Selects user-volume IDs by scope. |
engine/schema/src/main/java/com/cloud/storage/dao/VolumeDao.java |
Declares scoped user-volume selection. |
engine/schema/src/main/java/com/cloud/alert/ResourceAlertVO.java |
Maps fired-alert history records. |
engine/schema/src/main/java/com/cloud/alert/ResourceAlertRuleWebhookVO.java |
Maps rule-to-webhook associations. |
engine/schema/src/main/java/com/cloud/alert/ResourceAlertRuleVO.java |
Maps alert-rule configuration and ownership. |
engine/schema/src/main/java/com/cloud/alert/dao/ResourceAlertRuleWebhookDaoImpl.java |
Replaces and cleans webhook associations. |
engine/schema/src/main/java/com/cloud/alert/dao/ResourceAlertRuleWebhookDao.java |
Declares webhook-association operations. |
engine/schema/src/main/java/com/cloud/alert/dao/ResourceAlertRuleDaoImpl.java |
Implements rule searches, limits and precedence checks. |
engine/schema/src/main/java/com/cloud/alert/dao/ResourceAlertRuleDao.java |
Declares rule persistence queries. |
engine/schema/src/main/java/com/cloud/alert/dao/ResourceAlertDaoImpl.java |
Queries history, cooldowns and retention. |
engine/schema/src/main/java/com/cloud/alert/dao/ResourceAlertDao.java |
Declares alert-history persistence operations. |
engine/components-api/src/main/java/com/cloud/alert/ResourceAlertManager.java |
Defines the evaluator interface. |
api/src/test/java/org/apache/cloudstack/api/command/user/resourcealert/ResourceAlertRuleCmdTest.java |
Tests command error propagation. |
api/src/test/java/com/cloud/alert/ResourceAlertMetricTest.java |
Tests metric applicability and percentage classification. |
api/src/test/java/com/cloud/alert/AlertConditionTest.java |
Tests comparison operators and boundaries. |
api/src/main/java/org/apache/cloudstack/api/response/ResourceAlertRuleResponse.java |
Defines rule API responses. |
api/src/main/java/org/apache/cloudstack/api/response/ResourceAlertResponse.java |
Defines fired-alert API responses. |
api/src/main/java/org/apache/cloudstack/api/command/user/resourcealert/UpdateResourceAlertRuleCmd.java |
Exposes rule updates and webhook cleanup. |
api/src/main/java/org/apache/cloudstack/api/command/user/resourcealert/ListResourceAlertsCmd.java |
Exposes filtered alert-history queries. |
api/src/main/java/org/apache/cloudstack/api/command/user/resourcealert/ListResourceAlertRulesCmd.java |
Exposes filtered rule queries. |
api/src/main/java/org/apache/cloudstack/api/command/user/resourcealert/DeleteResourceAlertRuleCmd.java |
Exposes rule deletion. |
api/src/main/java/org/apache/cloudstack/api/command/user/resourcealert/CreateResourceAlertRuleCmd.java |
Exposes rule creation and ownership parameters. |
api/src/main/java/org/apache/cloudstack/api/ApiConstants.java |
Adds alert API parameter constants. |
api/src/main/java/org/apache/cloudstack/alert/ResourceAlertService.java |
Defines the alert API service contract. |
api/src/main/java/com/cloud/event/EventTypes.java |
Adds rule lifecycle event identifiers. |
api/src/main/java/com/cloud/alert/ResourceAlertRule.java |
Defines rule properties, types and states. |
api/src/main/java/com/cloud/alert/ResourceAlertMetric.java |
Defines supported metrics and resource applicability. |
api/src/main/java/com/cloud/alert/ResourceAlert.java |
Defines fired-alert properties. |
api/src/main/java/com/cloud/alert/AlertSeverity.java |
Defines alert severity levels. |
api/src/main/java/com/cloud/alert/AlertCondition.java |
Implements threshold comparisons. |
.github/workflows/ci.yml |
Adds resource-alert smoke tests to CI. |
Review effort: Balanced
| accountManager.buildACLSearchParameters(caller, cmd.getId(), cmd.getAccountName(), cmd.getProjectId(), | ||
| permittedAccounts, domainIdRecursiveListProject, cmd.listAll(), false); |
There was a problem hiding this comment.
This uses the same buildACLSearchParameters path as the other list APIs, so ResourceAdmin with listall behaves the same here as it does elsewhere. If that scoping needs changing, it would be a core change across all list APIs rather than in this PR.
| case NETWORK_READ_KBPS: { | ||
| if (type == ResourceAlertRule.ResourceType.Host) { | ||
| HostStats s = statsCollector.getHostStats(resourceId); | ||
| return s != null ? s.getNetworkReadKBs() : null; |
| ResourceAlertVO last = alertDao.findLastFiredForRule(ruleId, resourceId); | ||
| if (last == null) return true; |
| ResourceAlertRuleVO rule = findRuleForCaller(cmd.getId()); | ||
| checkEmailAccess(CallContext.current().getCallingAccount(), Boolean.TRUE.equals(cmd.getEmail())); | ||
|
|
||
| if (cmd.getName() != null) rule.setName(validateName(rule.getAccountId(), cmd.getName(), rule.getId())); |
| findRuleForCaller(cmd.getId()); | ||
| alertDao.removeByAlertRuleId(cmd.getId()); | ||
| return ruleDao.remove(cmd.getId()); |
| if (System.currentTimeMillis() - latest.getTimestamp().getTime() > maxAge) { | ||
| return null; | ||
| } | ||
| return gson.fromJson(latest.getVmStatsData(), VmStatsEntry.class); |
| sources.push(['StoragePool', 'listStoragePools', 'liststoragepoolsresponse', 'storagepool', {}]) | ||
| } | ||
| Promise.all(sources.filter(([, api]) => api in this.$store.getters.apis).map(([type, api, responseKey, itemKey, params]) => | ||
| getAPI(api, params).then(json => (json?.[responseKey]?.[itemKey] || []).map(r => ({ |
| const [api, responseKey, itemKey, params] = source | ||
| this.resourcesLoading = true | ||
| const hideExtensions = ['VirtualMachine', 'Host'].includes(this.form.resourcetype) | ||
| return getAPI(api, params).then(json => { |
| private String name; | ||
|
|
||
| @Parameter(name = ApiConstants.METRIC, type = CommandType.STRING, | ||
| description = "filter by metric, for example CpuUtilization") |
… network in resource alerts
|
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch. |
|
@blueorangutan package |
|
@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19963 |
…very management server collects
|
@blueorangutan test |
|
@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch. |
|
[SF] Trillian test result (tid-17543)
|
Description
This PR adds resource alerts. You set a rule on a VM, volume, host or storage pool, like "cpu above 80%", and CloudStack checks it every minute. When it goes over, an alert is fired and sent out.
New global settings: resourcealert.evaluation.interval, resourcealert.repeat.interval.default, resourcealert.history.retention.days, resourcealert.per.user.limit
Doc PR: apache/cloudstack-documentation#686
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Screenshots (if appropriate):
Rules list
Settings
New rule form
Rule details
Alert history of a rule
Alerts tab on a resource page
Alert received by a webhook
Alert deliveries on the webhook
Alert email
Alert on the event bus (RabbitMQ)
How Has This Been Tested?
Tested on a KVM lab with 2 management servers, 2 hosts and NFS storage. Went through the UI end to end, and checked everything with cmk and the management server logs.
Main thing tested was delivery, all three ways:
Also tested:
How did you try to break this feature and the system with this change?