Process/thread/greenlet safety

[dsuch]

Hello,

could you please tell me and document it if rlite is designed to be accessed from multiple processes or threads/greenlets?

I understand that in principle it's to Redis what SQLite is to network-based SQL servers but there is nothing in the documentation regarding that usage scenario.

Cheers.

[seppo0010]

Good idea, I will write about that. TL;DWY (too long, didn't write yet): it is multi-process safe, but not multi-thread safe.

[dsuch]

Cool, thanks - the application I have in mind (Zato) is in Python and uses both multiple processes and multiple greenlets within each processes. Each greenlet will count as a thread most likely?

If you could document which parts are thread/greenlet-safe, that would be great. Or, if none is, please document it as well.

Thanks a lot, that will be really helpful. I will certainly read it regardless of how long it is :-)

[seppo0010]

I don't know much about greenlet (that's an understatement, I just googled what it is), but I believe so.

If you open the connection on each thread/greenlet it will be safe.

Currently each command will lock the database, read the portion of it that needs, optionally write changes, and release the lock. Performing two concurrent commands would release the lock early and therefore any new data read may be invalid. Also the file descriptor will be unexpectedly closed. It can be made safer, but currently it is not.

[dsuch]

Right, I see - connections just cannot be shared between threads/greenlets without user's serializing access in their own code. Got it.

It's OK if you just add it to the docs instead of replying here but could you also please explain how it works with multiple processes when they discover that the DB is currently locked?

Also, because you said it is like SQLite, can you please mention that we discuss here pertains to each and every command, including read-only ones?

Thanks again.

[seppo0010]

I will write documentation with more details later. I'm just answering here so you don't have to wait until I have time to do that.

Currently opening a connection to a locked database will wait until the lock can be obtained. Do you have a use case for something like try_lock instead (meaning, return immediately an error when lock cannot be acquired)?

[dsuch]

Do you have a use case for something like try_lock instead

Not really, nope, though I imagine that library authors would like to have access to such a lower-level functionality. Me, I'd be just a user of the Redis-compatible API.

I run each group of Redis commands in separate greenlets so each can be blocked for however long is needed.

Now with rlite it could work similarly. I'm OK with opening a connection in each greenlet because this is only for the purpose of getting people quickly started with Zato without installing Redis proper. (As an aside, I can see that rlite-py is not API-compatible with redis-py so it's not a drop-in replacement but that's another story).

If on a typical laptop's HDD (7200 RPM) rlite can handle 50 new connections a second, each issuing 6-10 commands and closing, I will be happy :-)