SoftSignalBackend wrapping arbitrary callables

[burkeds]

Add getter, setter, and poll_period to SoftSignalBackend

Closes #1279

Motivation

Users working outside of EPICS/Tango — wrapping third-party Python APIs, calling analysis scripts, or interfacing with devices that have their own Python drivers — currently have no supported path to create signals without writing a full custom SignalBackend. This was raised at the Bluesky community workshop and is a known friction point for smaller lab-based groups.

Changes

Three new keyword-only parameters are added to SoftSignalBackend:

• getter: Callable[[], T | Awaitable[T]] — called on get_value and get_reading to fetch the current value from an external source. When poll_period is also set, called periodically while a subscription is active.
• setter: Callable[[Any], T | None | Awaitable[T | None]] — called on put in place of the default internal store update. May return the settled value the device actually reached; if it returns None and a getter is configured, the getter is called immediately to refresh the cache.
• poll_period: float — interval in seconds at which the getter is called while a subscription is active. Requires getter to be set.

All three parameters are optional. When none are provided, behaviour is identical to the existing SoftSignalBackend, so there are no breaking changes.

The internal self.reading store remains the single source of truth. The getter updates it rather than bypassing it, keeping subscriptions and cached reads coherent.

Modified soft_signal_rw and soft_signal_r_and_setter to take setter, getter, and poll_period arguments.

Design notes

• put is typed as Any to allow heterogeneous setter input types (e.g. a config class, a raw integer count, a command string) that differ from the signal's SignalDatatypeT.
• The polling task is started and cancelled in set_callback, tied to subscription lifetime.
• Transient getter failures during polling are swallowed and retried on the next interval, consistent with EPICS/Tango backend behaviour.
• get_setpoint deliberately does not call the getter — it returns the last requested value, not the live device readback.

[checkmarx-gh-ast-us-povs]

Checkmarx One – Scan Summary & Details – c601e52f-4034-4fab-b3e1-c208ec1a04ba

Great job! No new security vulnerabilities introduced in this pull request

---

Communicate with Checkmarx by submitting a PR comment with @Checkmarx followed by one of the supported commands. Learn about the supported commands here.

[oliwenmandiamond]

You will also need to update soft_signal_r_and_setter and soft_signal_rw to accept the new optional getter, setter and poll arguments to pass on to SoftSignalBackend

[jacopoabramo]

Looks good so far. Testing it locally with the pymmcore-plus package. It would have definetely saved me a lot of time.

I'm guessing some customization (such as changing the source) would still require subclassing but it's definetely an improvement

[jacopoabramo]

This PR only addresses procedural device declaration, right? Is a declarative approach achievable somehow?

[burkeds]

This PR only addresses procedural device declaration, right? Is a declarative approach achievable somehow?

Right, this addresses procedural only. Is a declarative approach even desirable in this case? Seems like you will likely need to write getters and setters anyway.

[coretl]

Hmm, is it a requirement for Signal.set to take a different datatype to its internal datatype? I know we do this for type coercion (take a string or a float, convert, store as float), but this example seems to promote this to requiring a different datatype. This will break typing...

[burkeds]

I don't think I quite understand. How does it promote requiring something?

If CallableSignalBackend.put calls any callable then it must be typed Any or perhaps T. This requires that Signal.set also be typed Any or T.

[coretl]

Example:

sig = soft_signal_rw(float, setter=config_setter)
await sig.set(32.4)  # pyright says ok, runtime says ok
value = await sig.get_value()  # value is float
await sig.set(MotorConfig(...))  # pyright says bad, runtime says ok
value = await sig.get_value()  # value is float

I don't think I want to promote a pattern where the set argument type is different from the value type

[burkeds]

@coretl What do you suggest then? If we say that the setter can only take one argument of the same type as the value carried by the function, then that strictly limits the types of operations we can reasonably support.

[jacopoabramo]

I have viewed a few third-party libraries for device control, not enough to constitute as an expert on the matter, but I haven't found so far the situation where an entire data structure is passed down to a function controlling a device. I've seen more use case of APIs with multiple parameters, but in that case using functools.partial to bind other arguments should be sufficient.

@burkeds have you encountered these situations before?

[burkeds]

    async def put(self, value: Any) -> None:
        write_value = self.initial_value if value is None else value
        self._setpoint = write_value
        ...

    async def get_setpoint(self) -> SignalDatatypeT:
        # For a soft signal, the setpoint and readback values are the same.
        if self._setter is None and self._getter is None:
            return self.reading["value"]
        else:
            return self._setpoint

Something like this?

[coretl]

Good point, we probably need the concept of setpoint and reading:

• initial value goes to self._setpoint and self.reading
• put value goes through converter, then stored as self._setpoint
• setter called on self._setpoint, if it returns a value that goes in self.reading, otherwise call getter to get that value

[burkeds]

A possible issue. Currently, the value is not converted until self.set_value(...).

If we put self._setpoint = self.converter.write_value(value) in set_value that works provided there is not setter. If there is a setter then the converted write value would not be accurate to assign as self._setpoint because the value is not converted before it is passed to the setter.

Should we pass the converted write value to the setter?

    def set_value(self, value: SignalDatatypeT):
        """Set the current value, alarm and timestamp."""
        self.reading = Reading(
            value=self.converter.write_value(value),
            timestamp=time.time(),
            alarm_severity=0,
        )
        if self.callback:
            self.callback(self.reading)

    async def put(self, value: Any) -> None:
        write_value = self.initial_value if value is None else value
        if self._setter is not None:
            written_value = await maybe_await(self._setter(value))
            if written_value is not None:
                self.set_value(written_value)
            elif self._getter is not None:
                await self._update_value_from_getter()
            else:
                self.set_value(write_value)
        else:
            self.set_value(write_value)

[coretl]

Should we pass the converted write value to the setter?

I think that's probably best

[burkeds]

Because the setter is possibly async, it can't go in set_value. That means to pass the converted value to the setter we need to convert the value twice, once in put and once in set_value. I didn't want to do that.

Instead, I thought we could pass the unconverted value to the setter (as I am sure a user would expect) and the setpoint is then recorded as the value passed to set_value.

[coretl]

This PR only addresses procedural device declaration, right? Is a declarative approach achievable somehow?

I think for declarative we should wait for the FastCS example. Passing setters and getters via a declarative approach gets messy fast...

[jacopoabramo]

Seems like you will likely need to write getters and setters anyway.

In my case, using pymmcore-plus, there is a Core object that does the heavy lifting in loading dedicated C++ DLLs and manages communication with the actual devices. I think that with declarative it would avoid some boilerplate code in having to create everytime the core object and dispatching it to a ... DeviceConnector, I think? From the declarative example the closest thing I can imagine is something like:

class MMCamera(StandardReadable, MMCoreDevice)
    ...

where core creation is delegated to MMCoreDevice. But then again for wrapping python packages with a more traditional API design this is fine so I'll just wait for the progress of FastCS, this works fine too

[coretl]

Assuming you're using AI to write this, please could you turn this into a "How to use soft signals" guide?

[burkeds]

The format is AI but otherwise I wrote it.

I renamed the file to "how to use soft signals". Did you want any other changes?

[coretl]

I meant put it in the docs/how-to, and add a section on using soft signals (without callables) as ways to store data. Then the rest of the document you wrote follows that says how to add callables to make the set/get have side effects and introducing soft command too.

[oliwenmandiamond]

The documentation above says that the setter method should be available here. Is this a mistake that it's left out?

[burkeds]

It was a mistake in the explanatory document. soft_signal_r_and_setter is meant to return a read-only signal so it does not take a setter argument. This has been corrected.

[burkeds]

@coretl For some reason, test_observe_value_times_out_with_no_external_task is seeing an elapsed time of 0.25s instead of ~0.07s consistently in python 3.13 but not in other versions. It is not clear to me why, some sort of change in how async tasks are scheduled?

If we relax the equality in the assertion to >= that would solve the issue but I am not sure if we want to accept this delay.

[coretl]

@coretl For some reason, test_observe_value_times_out_with_no_external_task is seeing an elapsed time of 0.25s instead of ~0.07s consistently in python 3.13 but not in other versions. It is not clear to me why, some sort of change in how async tasks are scheduled?

If we relax the equality in the assertion to >= that would solve the issue but I am not sure if we want to accept this delay.

I've noticed this on other PRs, I'll investigate...

#1295