Message Queue with per priority memory allocation by bevinduckett · Pull Request #5205 · nasa/fprime

bevinduckett · 2026-05-27T17:50:00Z


*Related Issue(s)*	#3603
*Has Unit Tests (y/n)*	Yes
*Documentation Included (y/n)*	Yes
*Generative AI was used in this contribution (y/n)*	Yes

Change Description

This PR adds:

Os::Generic::AtomicQueue: Lock-free MPMC circular buffer using atomic operations for O(1) enqueue/dequeue. ISR-safe (platform-dependent) with optional counting semaphore support for blocking operations. Serves as building block for PriorityMemQueue.
Os::Generic::PriorityMemQueue: ISR-safe, SMP-safe priority queue supporting up to 16 priority levels with per-priority AtomicQueue instances, independent memory pools, and static configuration system. Includes lock-free priority tracking via atomic bitmasks and rule-based unit tests
priority_buffer_analyzer.py: CMake-integrated Python script that parses topology dictionaries to auto-generate message size constants per component/priority, simplifying buffer sizing with automatic overhead accounting.

The code in this PR depends on #5204 and cannot be merged until that PR is complete

Rationale

F' allows a component to mark some ports as drop and some ports as assert on full. This allows "drop" ports to fill up the message queue and cause "assert" ports to fail to send a message leading to an assert. E.g. If the queue is filled up by messages that could be dropped, and then a single message comes in that cannot be dropped, ComQueue triggers an assertion:

Assert: "/workspaces/[...]/build-fprime-automatic-[...]/F-Prime/Svc/ComQueue/ComQueueComponentAc.cpp:947" 8

Adding a message queue with per priority memory allocation resolves that

   @ Port for receiving the status signal
      async input port comStatusIn: Fw.SuccessCondition priority 2

      @ Port array for receiving Fw::ComBuffers
      async input port comPacketQueueIn: [ComQueueComPorts] Fw.Com priority 1 drop

Separating messages by priority is also useful for ports with the same drop behavior when the different messages have different priorities (e.g. it would be bad for EventManager to drop a WARNING HI event because DIAGNOSTIC events consumed memory pool).

Testing/Review Recommendations

There's unit tests and PriorityMemQueue has been tested in a private deployment. I'd like to add it to a deployment in fprime-community but I'm not sure which

Future Work

F' components should be updated to use different priorities when sending messages with different urgencies or with different queue full behavior.

AI Usage (see policy)

Claude code - Sonnet 4.5. Used to assist with code generation under human supervision (careful review for the flight code, less careful review for the unit test code) and with documentation

+            fpm_types_values.BooleanType
+        )) 
+
+    def resolved_type_to_cpp_size_expr(self, resolved_type: fpm_types_values.Type, analysis: fpm_Analysis) -> str:


+    virtual ~PriorityMemQueue();
+
+    //! \brief copy constructor is forbidden
+    PriorityMemQueue(const QueueInterface& other) = delete;


+    PriorityMemQueue(const QueueInterface& other) = delete;
+
+    //! \brief copy constructor is forbidden
+    PriorityMemQueue(const QueueInterface* other) = delete;


github-advanced-security

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

…re conflicts Adopt the reviewed CountingSemaphore API from nasa#5204 (now merged into devel) and adapt PriorityMemQueue/AtomicQueue to the single-argument constructor. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…priority-queue

Copilot

Pull request overview

This PR introduces a new generic, priority-aware queueing implementation intended to prevent low-importance “drop” traffic from exhausting queue resources needed by higher-importance traffic (e.g., avoiding ComQueue asserts). It adds a lock-free Types::AtomicQueue as a building block, an Os::Generic::PriorityMemQueue that composes one atomic queue per priority, and a topology-driven Python analyzer to auto-generate per-component/per-priority buffer sizing constants.

Changes:

Add Types::AtomicQueue (lock-free MPMC circular buffer) plus unit tests and a design document.
Add Os::Generic::PriorityMemQueue (per-priority queues + priority tracking + blocking receive) plus extensive rule-based and validation unit tests.
Add priority_buffer_analyzer.py to generate PriorityBufferSizesAc.hpp from topology JSON artifacts.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`Os/Generic/Types/AtomicQueue.hpp`	Declares new lock-free atomic queue API with optional blocking enqueue support.
`Os/Generic/Types/AtomicQueue.cpp`	Implements atomic queue create/teardown/enqueue/dequeue/size helpers.
`Os/Generic/Types/test/ut/AtomicQueue/AtomicQueueTest.cpp`	Adds unit tests for basic and blocking behaviors of `AtomicQueue`.
`Os/Generic/Types/docs/sdd.md`	Adds SDD documenting the `AtomicQueue` algorithm and memory-ordering model.
`Os/Generic/Types/CMakeLists.txt`	Wires `AtomicQueue` into the `Os_Generic_Types` module and attempts to register its UT.
`Os/Generic/PriorityMemQueue.hpp`	Declares `PriorityMemQueue` and configuration structures/handle.
`Os/Generic/PriorityMemQueue.cpp`	Implements configured per-priority queues, send/receive, and config lifecycle.
`Os/Generic/test/ut/PriorityMemQueueTests.cpp`	Adds functional + concurrency + rule-based tests for `PriorityMemQueue`.
`Os/Generic/test/ut/PriorityMemQueueRules.hpp`	Adds STest rules used by `PriorityMemQueueTests.cpp`.
`Os/Generic/test/ut/PriorityMemQueueInputValidationTests.cpp`	Adds input validation tests for edge cases and invalid usage.
`Os/Generic/docs/sdd.md`	Extends OS Generic SDD with a new `PriorityMemQueue` section and usage notes.
`Os/Generic/DefaultPriorityMemQueue.cpp`	Adds an `Os::Queue` delegate implementation backed by `PriorityMemQueue`.
`Os/Generic/CMakeLists.txt`	Registers the new PriorityMemQueue implementation and unit tests.
`cmake/autocoder/scripts/priority_buffer_analyzer.py`	Adds topology analysis script to generate per-priority buffer size constants.
`.github/actions/spelling/expect.txt`	Adds spelling allow-list entries for new terminology (atomics, bitmask, MPMC, etc.).

+FwSizeType AtomicQueue::getSize() const {
+    // Safe to call on uninitialized queue
+    if (this->m_capacity == 0) {
+        return 0;
+    }
+
+    FwSizeType enq = this->m_enqueuePos.load(std::memory_order_relaxed);
+    FwSizeType deq = this->m_dequeuePos.load(std::memory_order_relaxed);
+    FwSignedSizeType diff = static_cast<FwSignedSizeType>(enq) - static_cast<FwSignedSizeType>(deq);
+    FW_ASSERT(diff >= 0, static_cast<FwAssertArgType>(enq), static_cast<FwAssertArgType>(deq));
+    FW_ASSERT(static_cast<FwSizeType>(diff) <= this->m_capacity, static_cast<FwAssertArgType>(diff),
+              static_cast<FwAssertArgType>(this->m_capacity));
+
+    // Approximate size (may be slightly stale in concurrent access)
+    return static_cast<FwSizeType>(diff);


+void PriorityMemQueueHandle::deallocateArrays(Fw::MemAllocator& allocator, FwEnumStoreType allocatorId) {
+    // Deallocate arrays in reverse order using stored allocator ID
+    FwSizeType arraySize = static_cast<FwSizeType>(this->m_maxPriority) + 1;
+    if (this->m_highWaterMarks != nullptr) {
+        // Explicitly destroy atomics before deallocation
+        for (FwSizeType i = 0; i < arraySize; ++i) {
+            this->m_highWaterMarks[i].~atomic();
+        }
+        allocator.deallocate(allocatorId, this->m_highWaterMarks);
+        this->m_highWaterMarks = nullptr;
+    }
+    if (this->m_atomicQueues != nullptr) {
+        // Explicitly destroy AtomicQueues before deallocation
+        for (FwSizeType i = 0; i < arraySize; ++i) {
+            this->m_atomicQueues[i].~AtomicQueue();
+        }
+        allocator.deallocate(allocatorId, this->m_atomicQueues);
+        this->m_atomicQueues = nullptr;
+    }
+    this->m_maxPriority = 0;
+    this->m_allocatorId = 0;
+}


+        // Get enabled priorities and non-empty hint mask
+        U32 enabledPriorities = this->m_handle.m_priorityMask.load(std::memory_order_acquire);
+        U32 nonEmptyHint = this->m_handle.m_nonEmptyMask.load(std::memory_order_acquire);
+
+        // Combine masks: only scan priorities that are both enabled and likely non-empty
+        U32 scanMask = enabledPriorities & nonEmptyHint;
+
+        // Scan priorities and attempt dequeue
+        if (scanAndDequeue(this->m_handle, destination, capacity, actualSize, priority, scanMask)) {
+            return QueueInterface::Status::OP_OK;
+        }


+void PriorityMemQueue::resetConfig() {
+    // Only call this in test environments after all queues are destroyed
+    if (s_configsUsed != nullptr) {
+        // Get allocator (same as used in config())
+        Fw::MemAllocator& allocator = Fw::MemAllocatorRegistry::getInstance().getAnAllocator(
+            Fw::MemoryAllocation::MemoryAllocatorType::OS_GENERIC_PRIORITY_QUEUE);
+        FwEnumStoreType allocatorId = 0;
+
+        // Deallocate the tracking array
+        allocator.deallocate(allocatorId, s_configsUsed);
+        s_configsUsed = nullptr;


thomas-bc · 2026-06-08T20:35:29Z


+## Os::PriorityMemQueue
+
+Os::PriorityMemQueue is an ISR-safe and SMP-safe, priority-based memory queue implementation for F´ using VxWorks message queues (msgQ). This implementation leverages the VxWorks msgQ API to provide robust multi-core synchronization without custom spinlock management, eliminating interrupt starvation issues while maintaining ISR safety.


Lots of notes about VxWorks which doesn't seem right - likely the docs were generated before AtomicQueue was introduced? We need to update

thomas-bc · 2026-06-08T20:43:29Z

+    this->m_bufferSize = bufferSize;
+    this->m_allocator = &allocator;
+    this->m_allocatorId = allocatorId;
+    this->m_id = allocatorId;  // Store for diagnostic logging


Is this using the same allocatorId on purpose ?

thomas-bc · 2026-06-09T00:53:09Z

+namespace Queue {
+// JPL heritage implementation supported up to 32 priorities per message queue, limiting to 16 to reduce memory
+// footprint
+constexpr static FwSizeType MAX_PRIORITIES = 16;


This should be clearly documented as it will limit usable priorities if we want to use standard components with PriorityMemQueue

Also need to remove the comment about JPL heritage

thomas-bc · 2026-06-09T01:05:52Z

+
+// Test configuration with single queue and single priority
+TEST(PriorityMemQueueConfig, SingleQueueSinglePriority) {
+    printf("\n=== TEST: SingleQueueSinglePriority ===\n");


Googletest can handle the logging, we shouldn't need to do any of that

thomas-bc · 2026-06-09T01:08:06Z

+    Os::Generic::PriorityMemQueue queue;
+
+    // Configure multiple priorities
+    Os::Generic::PriorityMemQueue::QueuePriorityConfig priorityConfigs[] = {{0, 128, 10}, {1, 128, 10}, {2, 128, 10}};


There's a lot of duplicated setup code - it'd be nice if AI could use a GTest fixture or helper functions to handle this and reduce the verbosity a bit

thomas-bc · 2026-06-09T01:35:44Z

+
+    //! \brief Constructor to initialize members
+    PriorityMemQueueHandle()
+        : m_atomicQueues(nullptr), m_maxPriority(0), m_notEmptySem(nullptr), m_highWaterMarks(nullptr) {}


Not all member variables are initialized properly. Could use the = 0/=nullptr on the definition rather than constructor init list

thomas-bc

Also I agree with your note in your PR, we really ought to be testing this in an actual deployment - we should tag up

bevinduckett added 5 commits May 27, 2026 03:36

Added Os::CountingSemaphore osal module (nasa#5173)

e2636bd

Added CountingSemaphore to SDD

e7f0338

Added new words for spell check of semaphore

5cdb769

Initial implementation of PriorityMemQueue using a lockless queue

8fd2214

Added flagged words to spelling & improved UTs

8a4e9b5

bevinduckett marked this pull request as draft May 27, 2026 17:50

bevinduckett and others added 2 commits May 27, 2026 10:51

Merge branch 'devel' into mem-priority-queue

4c55a7e

Merge branch 'devel' into mem-priority-queue

b15daaa

github-advanced-security AI found potential problems Jun 4, 2026

View reviewed changes

Comment thread cmake/autocoder/scripts/priority_buffer_analyzer.py

fpm_types_values.BooleanType

))

def resolved_type_to_cpp_size_expr(self, resolved_type: fpm_types_values.Type, analysis: fpm_Analysis) -> str:

github-advanced-security AI found potential problems Jun 4, 2026

View reviewed changes

thomas-bc and others added 3 commits June 5, 2026 22:02

Merge remote-tracking branch 'JPL-Devin/mem-priority-queue' into mem-…

07e4d39

…priority-queue

Resolve/revert devel merge quirks

5363467

thomas-bc marked this pull request as ready for review June 9, 2026 00:33

Copilot AI review requested due to automatic review settings June 9, 2026 00:33

Copilot started reviewing on behalf of thomas-bc June 9, 2026 00:34 View session

Copilot AI reviewed Jun 9, 2026

View reviewed changes

Minimal review changes and formatting

50486ca

thomas-bc requested changes Jun 9, 2026

View reviewed changes

thomas-bc reviewed Jun 9, 2026

View reviewed changes

devin-ai-integration Bot mentioned this pull request Jun 12, 2026

Fix PR #5205 CI failures: format, spelling, macOS portability, and PriorityMemQueue sanitizer/UT issues JPL-Devin/fprime#173

Open


		## Os::PriorityMemQueue

		Os::PriorityMemQueue is an ISR-safe and SMP-safe, priority-based memory queue implementation for F´ using VxWorks message queues (msgQ). This implementation leverages the VxWorks msgQ API to provide robust multi-core synchronization without custom spinlock management, eliminating interrupt starvation issues while maintaining ISR safety.

Conversation

bevinduckett commented May 27, 2026

Change Description

Rationale

Testing/Review Recommendations

Future Work

AI Usage (see policy)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-advanced-security AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thomas-bc Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

thomas-bc Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

thomas-bc Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

thomas-bc Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

thomas-bc Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

thomas-bc Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

thomas-bc left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants