speculo, or shared memory made easy
The last few months I've been dedicating most of my (very little) free time to a new project: speculo, a library which implements IPC (Inter-Process Communication) on top of shared memory. Since developers appreciate conciseness and minimalism, here's a description of speculo in a few bullet points:
At some point the garbage collector will kick in, when the conditions specified by the writer are met or when the SHM area is completely full. All the chunks which are still valid will be copied over to a new SHM object, and then the SHM objects will be atomically switched. Readers will be able to complete their ongoing reads, and transparently move on to the new SHM object as they request to read a new data chunk.
The above means that speculo can be used in at least two ways:
On the minus side, speculo doesn't implement any mechanism of change notification: when the writer writes a new data chunk, or updates an existing one, readers won't be notified of it (though they will get the new data, if they access the shared memory area). This was done intentionally, because the scope of speculo is just to make using shared memory a little easier, and to integrate with existing IPC mechanisms, such as sockets or D-Bus (or even new IPC mechanisms).
The project is in such a state where the declared functionality is working, though I wouldn't swear about its complete reliability (more tests are needed). The current API is not fixed in stone, and on the contrary I'm thinking of a couple of changes to apply. Also, after seeing the implementation of shm_open in Linux and FreeBSD, I'm seriously considering dropping the usage of it in favour of plain open, which would then allow for per-user namespaces. So, I think this is a good time to join the project, to shape it the way it can be useful to the largest number of people.
Last but not least, the link to the project page: https://gitlab.com/mardy/speculo
There is not much documentation in the wiki yet; I recommend checking out the code and running make doc to build the HTML documentation (requires doxygen). To run the tests, use make check (requires pkg-config and check).
A mailing list is up here.
- written in C
- POSIX (tested in Linux and FreeBSD)
- small (~850 LOC)
- well commented (~400 lines)
- good test coverage
- zerocopy
- lockless
- one writer, many readers
- data is written and read in chunks of arbitrary size
- a data chunk becomes visible to the readers as soon as the writer commits it
- data chunks can have an expiration time
- data chunks can be obsoleted by a newer copy
- garbage collector
- no file descriptors are permanently kept open
- no change notification
At some point the garbage collector will kick in, when the conditions specified by the writer are met or when the SHM area is completely full. All the chunks which are still valid will be copied over to a new SHM object, and then the SHM objects will be atomically switched. Readers will be able to complete their ongoing reads, and transparently move on to the new SHM object as they request to read a new data chunk.
The above means that speculo can be used in at least two ways:
- to implement a data stream: the writer fills the shared memory area with short-lived data chunks
- sharing changeable blocks of data
On the minus side, speculo doesn't implement any mechanism of change notification: when the writer writes a new data chunk, or updates an existing one, readers won't be notified of it (though they will get the new data, if they access the shared memory area). This was done intentionally, because the scope of speculo is just to make using shared memory a little easier, and to integrate with existing IPC mechanisms, such as sockets or D-Bus (or even new IPC mechanisms).
The project is in such a state where the declared functionality is working, though I wouldn't swear about its complete reliability (more tests are needed). The current API is not fixed in stone, and on the contrary I'm thinking of a couple of changes to apply. Also, after seeing the implementation of shm_open in Linux and FreeBSD, I'm seriously considering dropping the usage of it in favour of plain open, which would then allow for per-user namespaces. So, I think this is a good time to join the project, to shape it the way it can be useful to the largest number of people.
Last but not least, the link to the project page: https://gitlab.com/mardy/speculo
There is not much documentation in the wiki yet; I recommend checking out the code and running make doc to build the HTML documentation (requires doxygen). To run the tests, use make check (requires pkg-config and check).
A mailing list is up here.
Comments
There's also webmention support.