blog 06: new post - #12
Conversation
|
requested changes marked by line number |
nick-bors
left a comment
nick-bors
left a comment
There was a problem hiding this comment.
I just reviewed the writing; the typesetting I'm passing to Anthony.
Overall very interesting article and i think it deserves a space and i only really had questions (perhaps for future work) and minor re-wording comments. Good job. :)
| needs to be solved, we might as well resort to heuristics. Is this going to result in an exact solution? | ||
| Definitely not. But is this approach going to suffice in practice? Well, that seems very likely. | ||
|
|
||
| [^1]: Which is essential! Unless it is cyclic, there exists an algorithm with the time complexity of O(number of vertices). |
There was a problem hiding this comment.
- unless it is cyclic, there exists an ...This is worded strangely imo. Perhaps something
- Which is essential! Unless it is cyclic, there exists an algorithm with the time complexity of O(number of vertices).
+ Which is essential! There exists an algorithm for acyclic graphs with the time complexity of O(N).Also, try keep your notation consistent. Why is the above footnote not just O(N)? You already define it so why not use it? (also, super nitpick: whilst generally used as synonyms nodes and vertices aren't the same. Nodes have the possibility of holding data whilst vertices strictly do not. Using nodes here is more general and thus preferred)
|
|
||
| [^1]: Which is essential! Unless it is cyclic, there exists an algorithm with the time complexity of O(number of vertices). | ||
|
|
||
| [^2]: N means the number of nodes. Further down the article, we're going to be using this notation. |
There was a problem hiding this comment.
maybe
Where N means ...
would be more suitable.
Also this seems like something you would inline rather than make a footnote since you use it everywhere
|
|
||
| [^2]: N means the number of nodes. Further down the article, we're going to be using this notation. | ||
|
|
||
| [^3]: Which is not a lot! But that's enough unless we are looking for utterly perfect approximation, which we |
There was a problem hiding this comment.
you can remove utterly here
|
|
||
| ## Closing notes | ||
| Successful approximations from this network indicate that, in reality, when an NP-hard problem | ||
| needs to be solved, we might as well resort to heuristics. Is this going to result in an exact solution? |
There was a problem hiding this comment.
'might as well' here seems out of place. It seems like you have two things where one dominates the other (e.g is better in both aspects). That isn't the case -- from what ive gathered, only brute-force and approximations work in practice and perhaps you should highlight that here.
Consider something like:
'when an NP-hard problem needs to be solved, we must use heuristics in practice' which is more concise and a bit clearer
| As we can see, that's accurate enough. Keep in mind that the network finishes its calculations | ||
| practically instantaneously, immeasurably faster than a deterministic method. |
There was a problem hiding this comment.
would've been cool to benchmark this and throw it into an appendix. Maybe you would have seen some sort of pattern in both certainty and in speed depending on things like #of cycles, #of nodes, #of edges etc (you don't have to; its extra work. Its just what I was craving when reading this)
There was a problem hiding this comment.
also i am aware of the irony of me asking you to measure something you called 'immeasurably' faster.
... but just how fast is it really? like is that 10x, 100x, 1000x? what sort of speed-ups can you get from using an NN like this?
Perhaps all questions that you can answer in a follow-up
| - `GatedGraphConv` applies a shared Gated Recurrent Unit (GRU) cell across all nodes to recursively update their structural representations during neighborhood message passing. As I've seen GRU explained, | ||
| > Gated Recurrent Units (GRUs) use a reset gate and an update gate to regulate internal information flow. By selectively preserving or discarding historical state data at each step, GRUs mitigate the vanishing gradient problem and successfully capture long-term dependencies in sequential representations. | ||
|
|
||
| Essentially, what this gives us is that our 128 features might represent something like, "Am I a leaf?", |
There was a problem hiding this comment.
maybe you could emphasise the opaque nature of the internal representation a bit more with these features?
| "How close am I to a densely connected hub?", or any other characteristic. Also, because it uses 10 layers, | ||
| that means it learns stuff about friends, then friends of friends, and so on. | ||
| - `conv_norm` accepts the sum of the result of `GatedGraphConv` and the very original blueprint. | ||
| More precisely, it adds a tensor that is deterministically described by the original degree |
There was a problem hiding this comment.
tensor isnt that well known outside of NN stuff. perhaps another footnote?
|
|
||
| > ### TL;DR | ||
| > [Longest path problem](https://en.wikipedia.org/wiki/Longest_path_problem) has no known "fast" | ||
| > algorithm (unless [P=NP](https://en.wikipedia.org/wiki/P_versus_NP_problem)), but approximations suffice in practice. |
| Held-Karp's complexity grows as $$ \mathcal{\text{O}}(2^\text{N} \cdot \text{N}) $$. Which means that at 30 | ||
| nodes, the program would consume over 100 GB of RAM. | ||
| Of course, there are certain low-level optimizations. One example is capping the maximum number of nodes to 64 | ||
| and storing stuff inside a `uint64_t`. However, all those modifications don't render the program fundamentally faster. |
There was a problem hiding this comment.
Nitpick: I mean they would make it faster but it wouldn't change the inherent scaling characteristics. Its like a constant term in the big O. Also, as much as we drop constants in the big O notation, they really matter i mean what if this algorithm is best (if you're willing to sacrifice some ram) for smaller graphs? if at any point it would beat out brute force or the NN thats something that would be quite interesting
| Of course, there are certain low-level optimizations. One example is capping the maximum number of nodes to 64 | ||
| and storing stuff inside a `uint64_t`. However, all those modifications don't render the program fundamentally faster. | ||
|
|
||
| But what about heuristics? They can be accurate or very rough, fast or slow. Why not find a method to determine |
There was a problem hiding this comment.
nitpick: if you remove very it sounds nicer
3 participants
No description provided.