|
| 1 | +# Taurus Queue |
| 2 | + |
| 3 | +[](http://makeapullrequest.com) |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +Taurus Queue offers a comprehensive queue ecosystem, simplifying the creation, execution, management, and monitoring of scalable and highly available queues. Leveraging the robust foundation of the [Bull Project](https://github.com/OptimalBits/bull), Taurus Queue eliminates the intricacies of queue coding, allowing you to concentrate solely on your specific actions and rules, thus optimizing your time. It features a dedicated interface for efficient queue management and monitoring. |
| 8 | + |
| 9 | +## How the Queue Ecosystem Operates |
| 10 | + |
| 11 | +### Setup and Job Publishing |
| 12 | +1. Initialize your queues using this repository. |
| 13 | +2. Publish jobs to your queues using our publishers, compatible with multiple programming languages: |
| 14 | + - [PHP Taurus Queue Publisher](https://github.com/not-empty/taurus-publisher-php-lib) |
| 15 | + - [Go Bull Publisher](https://github.com/not-empty/taurus-publisher-golang) |
| 16 | + - Node.js (native support with examples included in this project) |
| 17 | + - Python (Currently in development) |
| 18 | + |
| 19 | +### Queue Management with Horus Taurus Manager |
| 20 | +3. Utilize [Taurus Manager](https://github.com/not-empty/taurus-manager-vue) for: |
| 21 | + - Pausing/unpausing, adding/removing jobs. |
| 22 | + - Deleting, retrying, debuggin, viewing error logs and much more. |
| 23 | + - Managing user permissions. |
| 24 | + - Overseeing your queues. |
| 25 | + |
| 26 | +### Real-Time Monitoring |
| 27 | +4. Implement [Taurus Monitoring](https://github.com/not-empty/taurus-monitoring) for real-time graphical insights of your entire ecosystem, integrating with [Grafana](https://grafana.com) and [Prometheus](https://prometheus.io): |
| 28 | + - Queue Length |
| 29 | + - Job Duration |
| 30 | + - Queue States |
| 31 | + - Failures by Queue |
| 32 | + - Total Jobs Completed (All-Time/Periodic) |
| 33 | + - Sum of Completed Jobs (All-Time/Periodic) |
| 34 | + |
| 35 | + |
| 36 | +### Installation |
| 37 | + |
| 38 | +[Release 1.0.0](https://github.com/not-empty/taurus-queue-nodejs/releases/tag/1.0.0) Requires [NodeJs](https://nodejs.org) 20.x |
| 39 | + |
| 40 | +When running Docker your queue will automatically be running and you just need to include your business rules. |
| 41 | +:) |
| 42 | + |
| 43 | +```sh |
| 44 | +docker-compose up |
| 45 | +``` |
| 46 | +### Setting Your Own Queue |
| 47 | + |
| 48 | +You can use our `default-business.js` located in `business` folder and include your rules / actions inside the try block |
| 49 | + |
| 50 | +```js |
| 51 | +... |
| 52 | +try { |
| 53 | +... |
| 54 | + // your actions and rules here |
| 55 | +... |
| 56 | +} catch (error) { |
| 57 | +... |
| 58 | +``` |
| 59 | +
|
| 60 | +Or you can create your own business (maybe multiple ones in same project, and even push from one queue to another one). |
| 61 | +For do that: |
| 62 | +
|
| 63 | +1 - Copy our `default-business.js` located in `business`, giving your name, in this sample we'll use `myown-business.js`; |
| 64 | +
|
| 65 | +2 - Change the name of the class on the top and the bottom of the file |
| 66 | +From this: |
| 67 | +```js |
| 68 | +... |
| 69 | +const BaseBusiness = require('./base-business'); |
| 70 | + |
| 71 | +/** |
| 72 | + * Example bussines job processor |
| 73 | + */ |
| 74 | +class MyOwnBusiness extends BaseBusiness { // changed from DefaultBusiness |
| 75 | +... |
| 76 | + |
| 77 | +module.exports = MyOwnBusiness; // changed from DefaultBusiness |
| 78 | +``` |
| 79 | +
|
| 80 | +3 - Include your rules / actions inside the try block |
| 81 | +```js |
| 82 | +... |
| 83 | +try { |
| 84 | +... |
| 85 | + // your actions and rules here |
| 86 | +... |
| 87 | +} catch (error) { |
| 88 | +... |
| 89 | +``` |
| 90 | +
|
| 91 | +4 - Last but not least declare your business in the `constructor.js` file inside the `config` folder |
| 92 | +```js |
| 93 | +const DefaultBusiness = require('../business/default-business'); |
| 94 | +const MyOwnBusiness = require('../business/myown-business'); // Added |
| 95 | + |
| 96 | +module.exports = { |
| 97 | + 'default': DefaultBusiness, |
| 98 | + 'myown': MyOwnBusiness, // Added |
| 99 | +}; |
| 100 | +``` |
| 101 | +### Running Your Queue Worker |
| 102 | +
|
| 103 | +Requires [Redis](https://redis.io/). |
| 104 | +
|
| 105 | +You can start with [Docker](https://docs.docker.com/get-docker/) using compose tool. |
| 106 | +
|
| 107 | +If you want to run the `default` queue just run the docker |
| 108 | +
|
| 109 | +```sh |
| 110 | +docker-compose up |
| 111 | +``` |
| 112 | +
|
| 113 | +You can now run your queue worker. (if you want to, just change the ./ops/docker/dev/run.sh to your new business, changing the "default" word for your queue name, in this case "myown") like this: |
| 114 | +
|
| 115 | +```bash |
| 116 | +#!/bin/sh |
| 117 | + |
| 118 | +npm i |
| 119 | +npm run dev myown 1 |
| 120 | +``` |
| 121 | +
|
| 122 | +You can enter in container e run a queue worker by yourself passing your queue name and the debug mode (1 or 0) |
| 123 | +
|
| 124 | +```sh |
| 125 | +docker exec -it taurus-queue bash |
| 126 | +``` |
| 127 | +
|
| 128 | +Running a queue worker in debug mode (in nodemon to update service when code changes) |
| 129 | +```sh |
| 130 | +npm run dev default 1 |
| 131 | +``` |
| 132 | +
|
| 133 | +Running a queue worker in debug mode |
| 134 | +```sh |
| 135 | +npm start default 1 |
| 136 | +``` |
| 137 | +
|
| 138 | +With the debug mode on, your queue will generate outputs from the `log.debug` command, make your development and debug easer. |
| 139 | +The `log.show` command always have output, so use carefully |
| 140 | +
|
| 141 | +You can also run a cluster with multiples queue workers, in this case we running 5 workers. |
| 142 | +* if you choose to do that, take care of your resources like Memory and CPU, it can be very, very heavy depends on the number of works and the operations of your business. |
| 143 | +
|
| 144 | +Running 5 queue workers in debug mode (in nodemon to update service when code changes) |
| 145 | +```sh |
| 146 | +npm dev-cluster default 5 1 |
| 147 | +``` |
| 148 | +
|
| 149 | +Running 5 queue workers in debug mode |
| 150 | +```sh |
| 151 | +npm start-cluster default 5 |
| 152 | +``` |
| 153 | +
|
| 154 | +### Pushing Jobs Itens On Your Queue |
| 155 | +
|
| 156 | +Now that you have your workers running, it's time to push itens to your queue. |
| 157 | +You can do that by running the `producer.js` or the `multi-producer.js` files in the example `folder`. |
| 158 | +As the name sujests, the `producer.js` send one job to the queue and the `multi-producer.js` send multiples. |
| 159 | +
|
| 160 | +In `producer.js` you can pass as parameter the name of the queue, if you not inform the `default` queue will be used. |
| 161 | +
|
| 162 | +You can also pass a JSON with your data. IF you not inform the JSON a default test data will be used. |
| 163 | +
|
| 164 | +If you want to run `producer.js` in the docker: |
| 165 | +
|
| 166 | +```sh |
| 167 | +docker exec taurus-queue node sample/producer.js default '{"data":"mydata"}' |
| 168 | +``` |
| 169 | +
|
| 170 | +If you inside the container or want to run locally: |
| 171 | +```sh |
| 172 | +node sample/producer.js default '{"data":"mydata"}' |
| 173 | +``` |
| 174 | +
|
| 175 | +In `multi-producer.js` you pass as parameter the name of the queue and the number of jobs. |
| 176 | +
|
| 177 | +If you not inform the name, the `default`queue will be used. |
| 178 | +
|
| 179 | +If you not inform the number of jobs it will use 2. |
| 180 | +
|
| 181 | +You can also pass a JSON with your data. IF you not inform the JSON a default test data will be used. |
| 182 | +
|
| 183 | +If you want to put 60 jobs on the default queue using docker: |
| 184 | +
|
| 185 | +```sh |
| 186 | +docker exec taurus-queue node sample/multi-producer.js default 60 '{"data":"mydata"}' |
| 187 | +``` |
| 188 | +
|
| 189 | +If you inside the container or want to run locally: |
| 190 | +```sh |
| 191 | +node sample/multi-producer.js default 60 '{"data":"mydata"}' |
| 192 | +``` |
| 193 | +
|
| 194 | +### Development |
| 195 | +
|
| 196 | +Want to contribute? Great! |
| 197 | +
|
| 198 | +The project using a simple code. |
| 199 | +Make a change in your file and be careful with your updates! |
| 200 | +**Any new code will only be accepted with all viladations.** |
| 201 | +
|
| 202 | +To ensure that the entire project is fine: |
| 203 | +
|
| 204 | +Run all validations |
| 205 | +
|
| 206 | +```sh |
| 207 | +$ npm run check |
| 208 | +``` |
| 209 | +
|
| 210 | +**Not Empty Foundation - Free codes, full minds** |
0 commit comments