issue requiring 'thread'

[rabidus]

I'm trying to require the 'thread' module in a lich script and keep getting an error whenever I run the script in lich. Is there something special that I need to do in order to be able to "require 'thread'" I do have ruby installed and it's in a windows environment. I can use threads all day in regular ruby scripts on my system, so I'm confused why it gives an error when I try to issue the command in a lich script

I believe the module thread relies on 'thread.so', which I saw Shaelun mention in an earlier post might cause an issue. However I've seen many posts with people talking about using threads, so what am I missing? Thanks!!

[BigWorm]

Heh, just had this issue the other day. Long story short, you probably don't actually need the thread library to what you'd like to do. Even on an SMP machine, the Ruby interpreter only actually spawns one thread, so any memory access operation is atomic.

Here's what Shaelun told me:

Well, that was a comically simple answer to what seemed like a hopelessly complex problem -- Lich doesn't include the C "thread" extension. I can't explain why you didn't get an error trying to ``require'' the thread extension (nor can I explain why i didn't), but after specifying the full path to a thread.so file, your code works exactly the same in Lich as it does in the shell.

Here's a copy/paste of my game window.


;l
--- Lich: keepalive, lichnet, mapper, calcredux, closecontainers.
;e echo $:
--- Lich: exec1 active.
[exec1: /usr/local/lib/ruby/site_ruby/1.8]
[exec1: /usr/local/lib/ruby/site_ruby/1.8/i686-linux]
[exec1: /usr/local/lib/ruby/site_ruby]
[exec1: /usr/local/lib/ruby/1.8]
[exec1: /usr/local/lib/ruby/1.8/i686-linux]
[exec1: .]
[exec1: gui]
--- Lich: exec1 has finished.
;e echo require("thread.so")
--- Lich: exec1 active.
--- Exception: no such file to load -- thread.so
--- Lich: exec1 has exited.
>;e echo require("/usr/src/ruby-1.8.6/.ext/i686-linux/thread.so")
--- Lich: exec1 active.
[exec1: true]
--- Lich: exec1 has finished.
;e start_script("/home/fallen/test.rb")
--- Lich: exec1 active.
--- Lich: test active.
1
--- Lich: exec1 has finished.
2
3
;l
--- Lich: keepalive, lichnet, mapper, calcredux, closecontainers, test.
4
;k
--- Lich: test stopped.

So, that's that. If you're aiming for compatibility, I really don't recommend doing it that way. It's obviously not flashy, but offhand the first thing that comes to mind is making sure spell-list.txt includes the relevant information, and having a simple script that's very little more than this:

[BigWorm]

FYI, that was in Linux, so the absolute path name will probably be different for you (that's why we had talked about compatability).

If you tell me what you are trying to do, I can probably help you come up with a workaround for this.

[rabidus]

Although I realize that ruby threads aren't native, I believe they are well suited to do asynchronous message passing. I'm doing some cross character communication stuff among other things. I'll have to try requiring the .so file directly like you suggested. Sounds like a good workaround to me.

I do have a question about files such as crosscharcomm.lic that don't require or load any modules, but still are able to use Thread and Socket classes. How does that work? Is there some sort of environment setup that happens when lich loads?

Thanks for the help!

[Shaelun]

Although I realize that ruby threads aren't native, I believe they are well suited to do asynchronous message passing. I'm doing some cross character communication stuff among other things. I'll have to try requiring the .so file directly like you suggested. Sounds like a good workaround to me.

I do have a question about files such as crosscharcomm.lic that don't require or load any modules, but still are able to use Thread and Socket classes. How does that work? Is there some sort of environment setup that happens when lich loads?

Thanks for the help!

There's no need to include Ruby's "thread" library to use threads. They're an integral part of all interpreter distributions (at least all the ones I've seen, including the best "unofficial" virtual machine implementation I know of, YARV [it basically turns Ruby into a bytecode compiled language like Java]). As for sockets, that extension is statically linked in to the executable -- it's already "required" so to speak.

The standard thread library provides condition variables, mutexes, and... well probably some other random stuff, but I've never had a reason to use it, so I dunno. It's primarily for advanced thread synchronization.

Every script that Lich is running is actually its own Ruby thread -- there's nothing stopping a script from forking off additional threads though. There's a couple of things you'll want to bear in mind while doing that though...

First, things like the get method will consume data, so if multiple threads are using it, they'll be stealing lines from eachother.

Second, a script isn't actually paused immediately upon the user pausing it -- it's actually only paused when it attempts to interact with the game. What that means is that if you fork off a thread that's looping and incrementing a variable or something, pausing the script will have no effect on it.

Third, you'll get into trouble if you start changing the thread groups around. Lich uses them to associate all threads to a given script, so that it knows which ones to kill if the script is killed.

It may be helpful to know that I generally followed the same policy that Linux seems to -- Lich won't stop you from doing stupid things, because that would stop you from doing clever things too (you can easily shoot yourself in the foot if you aren't paying attention).

If you don't want to deal with any of this, you can always just let Lich handle it all like it usually does and use multiple scripts the same way you'd use multiple threads. If that doesn't cover everything you wanted to know, just say so and I'll elaborate.

[rabidus]

Is there any way that you know of to include the standard ruby thread library so I can have access to mutexes etc? When I tried requiring the full path to the thread.so library it still did not recognize the Mutex keyword. Any ideas?

[BigWorm]

If you're on linux, you could always used a named pipe for IPC if its producer-consumer model.

[Shaelun]

Is there any way that you know of to include the standard ruby thread library so I can have access to mutexes etc? When I tried requiring the full path to the thread.so library it still did not recognize the Mutex keyword. Any ideas?

Truthfully I'm really not sure what you could possibly want to do that requires mutex locks... if you use an IO driven model, all the coordination you could need with multiple characters will take care of itself. Still, far be it from me to say you shouldn't play around with stuff -- you can use the pure Ruby version of the thread extension; I'm attaching the latest copy.

FYI, you need to know that Ruby implements all of this functionality through critical sections, and at least half of Lich is executing in the context of Ruby threads -- meaning that if you go and do something like pause in the middle of a critical section, Lich is paralyzed until you release that lock. You won't be able to kill the script, release the lock, or do anything else until that thread exits the critical section.

In short, make sure you watch out for possible deadlocks, or you're going to be forcibly terminating the Lich process and logging back into the game a whole lot.

[Shaelun]

Although I realize that ruby threads aren't native, I believe they are well suited to do asynchronous message passing.

I'll be the first to admit my limitations, and after pondering what you could possibly be trying to do after making my last post, I realized I was a little fuzzy on exactly what you meant by "asynchronous message passing." So I looked it up, and if I understand what "message passing," has come to mean these days (i.e. distributed computing), you want to share data between processes on different physical machines...? If so, I'm completely lost.

If that's the case, how would a mutex on one machine have any influence whatsoever on what a different physical machine is doing...? Wouldn't any implementation of this by definition have to be input-output driven, and most easily made thread-safe by using blocking socket operations?

[rabidus]

Just to fill ya in on why I was asking about mutexes....

The idea is to build a server that client characters can connect to that directs traffic so to speak for the entire group. The server I'm envisioning would need to keep track of clients, and would need to be asynchronously sending/receiving requests out to clients as they are made. This doesn't really require anything of lich since the server would be run on a completely separate process. However the clients would need to be run from within the Lich environment, and I envision them running asynchronously as well. The following is a specific case in which I was wanting to use mutexes: the server issues some command to the client and the client executes the command (which may take a while), and then something happens and the server wants to request that the client cancel the command. Well the client would have to be looking for more commands from the server at the same time (logically, I realize these are 'green' threads) as the first command is being executed in case there is a cancellation request. One way of keeping track of which commands are being executed at any given time requires some sort of data structure that has a list of all the threads and the associated command that the thread is executing. At the very end of the thread/command execution, you would want to remove that entry from the data structure signifying it is no longer active. This act of removing an entry from the data structure would come from the child thread just before it exits. The parent thread would be adding new entries to the structure every time a new command is issued. Having two different threads modifying the same data structure without any thread synchronization (mutexes) can lead to unexpected results.

I've since thought of other ways to get around this particular problem, however the use of mutexes would be needed any time there are multiple threads modifying the same variable (which should be avoided if possible).

I appreciate the community here, and how helpful people have been. If I ever get anything to the point that I think other people might be able to use it I'll definitely post it up here.

On a similar note, is the script repository still being maintained? I haven't had time to check for myself, but that seems like a great place to go poke around in. What are some of you guys' favorites from the repository that aren't included with the windows install?

[BigWorm]

What we were trying to tell you is that all memory operations are atomic when they are accessed by threads running on the same interpreter.

[rabidus]

I think a lot of methods in commonly used data structures make multiple calls to memory. Although each call to memory is atomic, there is a need to synchronize the method calls that make multiple calls to memory (which Array.push does).

I could be mistaken here, but this is how I understand it to work. Say Array.push requires 4 calls to memory, and Array.drop requires 2. If one thread calls push while the other calls drop on the same variable at nearly the same time the execution could end up looking like this

push call 1
drop call 1 <- this call could invalidate assumptions that "push" has made by changing the internal state of the variable
push call 2
drop call 2
push call 3
push call 4

Where what you would want was this

push call 1
push call 2
push call 3
push call 4
drop call 1
drop call 2

[Shaelun]

I think what you're referring to is the basic caveat of threads, rabidus. If you synchronize threads to that degree, you basically lose any advantage you'd be getting from them -- ultimately you'd end up doing a lot more work than you would if you just got the job done with a single thread. What I guess I'm saying is that if your threads share that much data, maybe they shouldn't really be executing in the context of multiple threads to begin with (obviously that's not universally true though).

Whenever a thread attempts to access an array that's currently being modified by another thread, the thread attempting access raises (throws, whatever you call it) a RuntimeError. I'm still not sure I really get what you're talking about doing, but wouldn't this work just fine...? It's not exactly robust, but it's functional (and dreadfully insecure...):



server = TCPSocket.new(serverAddress, serverPort)
CANCELLATION_ORDER = "some unique cancel msg"
cmdAry = []

loop {
cmd = server.gets
if cmd == CANCELLATION_ORDER
begin
cmdAry.pop.kill
rescue NoMethodError
fail "Server tried to cancel an operation when there isn't one to cancel..."
rescue RuntimeError
retry
end
else
cmdAry.push( Thread.new {
eval(cmd)
begin
cmdAry.delete(Thread.current)
rescue RuntimeError
# The `cmdAry' array was being modified at that moment... retry the operation
retry
end
})
end
}

BTW, I hate to contradict you BigWorm, but technically the operations aren't atomic (to get really technical, it actually depends on what platform Ruby is running on and what thread library it's using...). You can usually pretend they're atomic and get away with it just fine though, because Ruby is so friendly it actually watches for complications like that and takes care of it itself (again, usually -- it's not perfect). For instance, there are only a couple of places in all of Lich I had to go low-level and make sure concurrent threads didn't cause problems (which I find pretty impressive, since Lich spawns a dozen or more Ruby threads when you get a handful of scripts going).

This isn't a programming forum, but since it may be of interest to you, what Ruby does is check for a viable threading library -- if the current platform can't provide one, it resorts to its internal threading code. Basically what that amounts to is Ruby installing a timer that sends a specific signal to the process every 10 microseconds or so (take that number with a big grain of salt). When Ruby receives said signal, it uses either setjmp or getcontext to store all the data it's going to need to pick up where it left off from, then calls its thread scheduler, which sets the currently executing thread. Ruby then uses the previously saved context for whatever thread is now flagged as the "currently executing" one to jump back to what it was doing for another 10 microseconds, ad infinitum. I think it's a simple round-robin policy, but I'm not sure if my memory about that is accurate.

Ideally, though, a pthread (POSIX thread) library is available, and Ruby can make use of threading code that allows the OS kernel to do things like pre-empt an executing thread when necessary. Threading gets extremely low-level... having some experience in assembly language really helps clarify things, but just bearing in mind that you can control the execution of a process in detail by manually setting the CPU registers should be enough to understand what an execution context means (it's basically just a snapshot of the CPU registers at a given time).

You'll have to pardon me if none of this comes as news to either of you; I'm not trying to patronize anyone. By the by, I'm not involved in the development of Ruby... I just studied the code extensively -- so don't make the mistake of thinking this info is infallible or anything :)

[BigWorm]

I'm pretty sure Ruby still uses green threads as of version 1.8 and 1.9 only supports native threads for some things in Unix, but the Windows version will be green threads still. The global interpreter lock will pretty much prevent you from shooting yourself in the foot at least with scalars, arrays, and hashes.

[Shaelun]

You may well be right, and ultimately since most people use Lich in Windows, it's pretty much a moot point.

There's a number of places in Ruby's eval.c where it makes reference to native threads, which is why I came to the conclusions I did. For example...



void
ruby_init()
{
static int initialized = 0;
static struct FRAME frame;
static struct iter iter;
int state;

if (initialized)
return;
initialized = 1;
#ifdef HAVE_NATIVETHREAD
ruby_thid = NATIVETHREAD_CURRENT();
#endif

The only documentation I've seen is from Ruby 1.82, so I figured it was outdated. Again though, you may well be right; to be honest, aside from academic curiosity, it doesn't make much of a difference to me and my outdated single core CPU :)