v0.3.2.2-beta - [UPDATE] - Big improvements to convergence tracking (more coming soon), implemented "sticky tokens" and "lead callers" for secure data locality and cache isolation. See release_notes.md for further details. ADDED missing critical testing file!!! (Sorry everyone for not noticing the index to my client demo, all demos should be working now.)

Author: TavariAgent

CONTRIBUTING.md

@@ -4,14 +4,18 @@ Thanks for your interest in TokenGate.
 
 ---
 
-This project is currently shared primarily as a proof of concept.  
-Feel free to clone, fork, and experiment with the code.
+TokenGate is an open-source, open-contribution project.   
 
-If you'd like to contribute:
+Contributions are welcome in the form of bug fixes, new features, documentation improvements, and  
+any changes that can enhance the usefulness or performance of the system. 
 
-- Open an issues when problems arise
-- Keep contributions focused and small
-- Prefer clarity over cleverness
-- Preserve research tone of the project
+To contribute to TokenGate please follow these steps:
+1. Fork the repository and create a new branch for your changes.
+2. Make your changes and ensure that they are well-documented and tested.
+3. Submit a pull request with a clear description of your changes and what they solve.  
 
-Bug reports, doc clarifications, system modifications, all contributions are welcome.
+Please ensure that your code adheres to the existing style and conventions of the project.  
+
+Contributions will be reviewed and merged based on their quality, relevance, and alignment with the project's goals.  
+
+Thanks for helping to make TokenGate better!

DOCS/BETA.md

@@ -2,44 +2,36 @@
 
 ## Overview
 
-Welcome to the TokenGate Beta documentation! This document provides an overview of the current state of the  
-TokenGate project, including its features, limitations, and future plans. I also include a quick startup guide  
-for both the WebSocket and regular operations. Please note that this is a beta version, and while it is  
-stable and actively tested, edge cases and rough spots are expected at this stage.  
+Welcome to the TokenGate Beta documentation!   
 
+This document provides an overview of the current state of TokenGate.
 
 ### Current Features
 
 - **Token based concurrency:** The core feature of TokenGate is its token-based concurrency model, which  
-    allows for efficient coordination of tasks across multiple threads.  
+    allows for efficient coordination of tasks across multiple threads.
 - **WebSocket support:** TokenGate includes support for WebSocket communication, enabling real-time  
     monitoring and control of tasks.
-- ***DoS protection*:** The token system includes built-in DoS protection ("auto-blocking") to prevent overwhelming    
+- **DoS protection:** The token system includes built-in DoS protection ("auto-blocking") to prevent overwhelming    
     the system with too many "failed" concurrent tasks. (Tasks which do not return their result are "failed".)
 - **Telemetry:** TokenGate provides telemetry data for tasks, allowing for monitoring and debugging of concurrent   
     operations with clarity.
+- **Flexible API:** The API is designed to be flexible and easy to use, allowing developers to quickly integrate  
+    TokenGate into their applications with minimal setup.
+- **Token safety:** The system is designed to prevent data loss and ensure that tokens are properly managed  
+    based on hashing as well as controlled token locality.
 
 ### Limitations
 - **Beta status:** As a beta project, TokenGate may have bugs or performance issues that have not yet been    
     identified or resolved.
-- **Limited documentation:** The documentation is currently limited and may not cover all aspects of the    
-    project in detail. I plan to expand the documentation as the project progresses.  
+- **Limited documentation:** The documentation is currently limited and does not cover all aspects of the    
+    project. I plan to expand the documentation as the project progresses.  
 - **Performance under extreme load:** While TokenGate is designed to handle a high volume of tasks, its   
-    performance under extreme load conditions needs long term analysis.  
-
-### Future Plans
-- **Improved documentation:** I plan to expand the documentation to provide more detailed information on how    
-    to use TokenGate and its various features.
-- **Performance optimizations:** I will continue to optimize the performance of TokenGate, especially under  
-    high load conditions.
-- **Additional features:** I am considering adding additional features such as task control mechanisms like  
- "reset token status" to adjust auto-blocking live and possible dristributed compute support in the future.
+    performance under extreme load conditions needs long term analysis.
 
 ## Get Started
 
-### Detailed startup guide:
-
-#### Using OperationsCoordinator without WebSocket:
+### Using OperationsCoordinator event bus *without* WebSocket:
 ```python
 from operations_coordinator import OperationsCoordinator # Must accompany main()
 from token_system import task_token_guard # The independent imported decorator for your functions
@@ -68,28 +60,81 @@ def string_operation_task(task_data):
 # Setting up an IO-centric method with the token decorator:
 # IO writer counts for 'storage_speed':
 # 'SLOW' (10 writes), 'MODERATE'(25 writes), 
-# 'FAST' (50 writes), 'INSANE' (70 writes) <- CAUTION
-@task_token_guard(operation_type='data_processing', tags={'weight': 'heavy', 'storage_speed': 'MODERATE'})
+# 'FAST' (50 writes), 'INSANE' (70 writes) <- CAUTION only top-end SSDs will handle this.
+@task_token_guard(
+    operation_type='data_processing', 
+    tags={'weight': 'heavy', 'storage_speed': 'MODERATE'}
+)
 def data_processing_task(task_data):
     # Simulate a data processing task
     return result
+
+
+# -----------------------------------NEW*-----------------------------------
+# As of v0.2.2.0-beta Awaiting tokens is supported, allowing for more complex
+# orchestration patterns. "result = await token" works to await a single token.
+
+# Or a full batch can be awaited with asyncio.gather:
+async def main():
+    coordinator = OperationsCoordinator()
+    coordinator.start()
+    try:
+        tokens = [my_task(i) for i in range(64)]
+        results = await asyncio.gather(*tokens)
+    finally:
+        coordinator.stop()
+
+asyncio.run(main())
+
+# -----------------------------------NEW------------------------------------
+# A "sticky_anchor" tag can be added to any decorator to give the sticky 
+# key an explicit name, independent of operation type. Sticky tokens are 
+# pinned to the same core domain and create a path for related operations 
+# to follow the same route, ensuring data locality and consistent performance
+# even when the operations do not require this behavior to work.
+
+# This happens automatically when the system interprets tokens with matching
+# (operation_type, args) keys.
+@task_token_guard(
+    operation_type="my_op", # Use a unique operation type for clarity.
+    tags={"weight": "medium", "sticky_anchor": "op_token"}, 
+    # The 'sticky_anchor' tag gives this token a specific known moniker.
+    # Good naming conventions for the sticky_anchor value can assist debugging.
+)
+def my_operation(n: int) -> int:
+    ...
+
+# -----------------------------------NEW------------------------------------
+# "Lead tokens" — those decorated with 'external_calls' generate a SHA-256 seed from their 
+# token ID and call list. That seed is pinned to a core domain. Any token spawned during 
+# the lead's execution inherits the seed and gets routed to the same core automatically. 
+# The domain releases when the lead and all of its children have completed. This is a 
+# production ready feature that provides deterministic routing, prevents data races, 
+# and measurably better behaviour under saturated load conditions for complex call chains.
+@task_token_guard(
+    operation_type="lead_op",
+    tags={"weight": "medium", "external_calls": ["child_op"]},
+)
+def lead_operation(n: int) -> list:
+    return [child_op(n + i) for i in range(4)]
 ```
 
-#### Running the WebSocket server:
+### Running the WebSocket server:
+
 ```bash
 python launch_gui.py OR python -m your_package_root.launch_gui
 ```
 *(Note: Adjust the path if running from outside the repository root).*
 
 
-#### Using OperationsCoordinator with WebSocket:
+#### Using OperationsCoordinator *with* WebSocket:
 ```python
 from operations_coordinator import get_global_coordinator # Must accompany main()
 
 # Get coordinator (GUI already started it)
 get_global_coordinator() # One line at module-level to rule the whole WebSocket!
 
-# No specific calls for main using the WebSocket
+# No specific calls for main using the WebSocket.
 def main():
     # Main is free now!
     try:
@@ -98,7 +143,7 @@ def main():
         pass
 
 if __name__ == "__main__":
-    main() # This gets used by the entry point in the WebSocket launcher to route 'func' (function)!!!
+    main() # This gets used by the entry point in the WebSocket launcher to route 'func'!
 
 # Decorated functions will use task_token_guard and the WebSocket integration will   
 # detect when the coordinated main is running to track task submissions.
@@ -126,8 +171,6 @@ When you first set up your WebSocket environment and want to run a task, you sho
 - **Task Launcher:** Your registered task should appear in the task launcher with its description and category.
 ![Task Launcher](/assets/task_launcher.png)
 
-
-
 - **Controls:** When you run the task, there is controls where you can monitor and manage it (e.g., stop, restart).
 ![Admin Dashboard](/assets/per_operation_controls.png)  
 
@@ -137,8 +180,9 @@ When you first set up your WebSocket environment and want to run a task, you sho
 
 ## Conclusion
 
-TokenGate is an active beta project that aims to provide a powerful and flexible concurrency management system   
-using tokens. While it is currently functional, there are still areas for improvement and optimization. I encourage   
-users to explore the project, provide feedback, and contribute to its development. 📦 🚀
+TokenGate is an active beta project that aims to provide a powerful and flexible production ready 
+concurrency management system using tokens. While it is currently functional, there are still areas for 
+improvement and optimization. I encourage users to explore the project, provide feedback, and contribute 
+to its development. 📦 🚀
 
-To quickly reach out to the developers go here and drop a message (all forms of feedback are accepted): [Tavari](https://www.tavari.online)
+To quickly reach out go here and drop a review in the feedback section: [Tavari](https://www.tavari.online)

DOCS/SETUP.md

@@ -24,9 +24,8 @@ class OperationsCoordinator: # Under this class are where you find configuration
             self, 
             workers_per_core: int = 4, # Do not modify this without disabling convergence! Do not go under 2 workers.
             enable_convergence: bool = True, # Controls dynamic worker counts.
-            convergence_verbose: bool = False, # Now lives in tg_print!! Controls verbosity, DO NOT enable with REPL.
-            base_memory_budget_mb: int = 45, # This is a control for operation memory (adjustable).
-            num_executors: int = 8, # This controls number of parallel executors.
+            base_memory_budget_mb: int = 45, # This is a control for warming up operation memory locals (adjustable).
+            num_executors: int = 8, # This controls number of parallel async executers (for token gathering).
             auto_block_dangerous: bool = False, # Blocks explosive tasks with possibly dangerous inputs.
     ):
 ```

DOCS/proof-of-concept.md

@@ -673,46 +673,46 @@ def record_task_routed(self, core_id: int, weight: TaskWeight):
         self.total_routed += 1
         self._affinity_counts[core_id][weight.value] += 1    
 
-# Async gathers task tokens setting them to the correct state for execution by workers.
-    async def _execute_token(self, token: TaskToken, worker_id: str, core_id: int):
-        """Execute one admitted token on its already-selected core path.
-
-        This method performs the lifecycle transition to EXECUTING, runs the
-        wrapped callable through the executor-backed path, stores the result or
-        error on the token, records execution history for the coordinator, and
-        triggers retry/Guard House hooks when configured.
-        """
-        # Transition to executing
-        if not token.transition_state(TokenState.EXECUTING):
-            print(f"[{worker_id.upper()}] Failed to transition {token.token_id}")
-            return
-
-        start_time = time.time()
-        success = False
+# Async gathers task tokens setting them to the correct state for execution with workers.
+async def _execute_token(self, token: TaskToken, worker_id: str, core_id: int):
+    """Execute one admitted token on its already-selected core path.
+
+    This method performs the lifecycle transition to EXECUTING, runs the
+    wrapped callable through the executor-backed path, stores the result or
+    error on the token, records execution history for the coordinator, and
+    triggers retry/Guard House hooks when configured.
+    """
+    # Transition to executing
+    if not token.transition_state(TokenState.EXECUTING):
+        print(f"[{worker_id.upper()}] Failed to transition {token.token_id}")
+        return
 
-        try:
-            loop = asyncio.get_running_loop()
-            # This is now "partial" instead of lambda for better stability.
-            bound_func = partial(token.func, *token.args, **token.kwargs)
-            # We use run_in_executor to execute the task in a thread, 
-            # allowing us to manage concurrency and avoid blocking the event loop.
-            result = await loop.run_in_executor(None, bound_func) 
-            token.set_result(result)
-            self.total_executed += 1
-            success = True
-
-            print(f"[{worker_id.upper()}] ✓ Completed {token.token_id}")
+    start_time = time.time()
+    success = False
 
-        except Exception as e:
-            # Failed!
-            token.set_error(e)
-            self.total_failed += 1
-            if self.result_verbose:
-                print(f"[{worker_id.upper()}] ✗ Failed {token.token_id}: {e}")
-
-        finally:
-            execution_duration = time.time() - start_time
-        # Additional execution recording and Guard House checks would go below here.
+    try:
+        loop = asyncio.get_running_loop()
+        # This is now "partial" instead of lambda for better stability.
+        bound_func = partial(token.func, *token.args, **token.kwargs)
+        # We use run_in_executor to execute the task in a thread, 
+        # allowing us to manage concurrency and avoid blocking the event loop.
+        result = await loop.run_in_executor(None, bound_func) 
+        token.set_result(result)
+        self.total_executed += 1
+        success = True
+
+        print(f"[{worker_id.upper()}] ✓ Completed {token.token_id}")
+
+    except Exception as e:
+        # Failed!
+        token.set_error(e)
+        self.total_failed += 1
+        if self.result_verbose:
+            print(f"[{worker_id.upper()}] ✗ Failed {token.token_id}: {e}")
+
+    finally:
+        execution_duration = time.time() - start_time
+    # Additional execution recording and Guard House checks would go below here.
 ```
 
 ### Mailboxes: