New Hardware and status on lag and invoker

[Rinualdo]

http://forums.play.net/forums/GemStone%20IV/GemStone%20IV%20Announcements/Important%20Announcements/view/3791

Found the invoker interesting
We are getting closer to our new servers.

https://i.imgur.com/m76G1xY.png

We're looking at tentatively March 31st for when the swap will take place. We'll post for sure when that's a bit more locked in, as there will be significant downtime (between 4 and 8 hours as long as it all goes smoothly.

I just want to take a moment to go over everything we have done in the last few weeks. We've been monitoring our systems, which create a little bit of lag, but we've been able to find areas where the game was bogged down. We've tweaked several major systems and introduced the invoker so that we could measure metrics on our spells, as this was one of the areas we tried to improve performance.

We increased efficiency in a number of areas that reduced strain (which resulted in lag) on the game. There are still areas needing improvement, but we're a long ways from what we were seeing in January and February. I realize that everyone has different experiences, but strictly talking about in-game lag, it's not happening as often as it was. There are a number of areas where lag will still occur. Rooms with many players, the command queue, movement, and the room window. Localized lag is a thing, like the front end crashing or freezing. There were also a number of ISP related issues this month that cropped up. We also had some server work going on (the Store, website, etc). We will continue to work on these areas where we can to improve the game for everyone.

As for the invoker, it was put out, unannounced, to see how it would go. This being out all the time is not a permanent feature. It will move toward a schedule when it comes around and possible only be around for special cases. It will not be anything behind the SimuCoin Store.

I'll post more across our platforms (and email) when we know more about our scheduled downtime.

[Tgo01]

As for the invoker, it was put out, unannounced, to see how it would go. This being out all the time is not a permanent feature. It will move toward a schedule when it comes around and possible only be around for special cases. It will not be anything behind the SimuCoin Store.

At least they made an announcement on the invoker. I think the lack of an announcement is what was driving most people crazy.

I think they could have also avoided a lot of ill will if they just came right out and said all of this from the very beginning: There are many issues with the game, we are trying to improve those issues, and we will monitor the changes we made to see how things work out and see if there is still room for improvement.

Simutronics seems to have a long history of getting it wrong the first time but eventually getting it right.

[Taernath]

Didn't they upgrade servers a couple years ago?

I guess we'll see how this plays out. If the core issue is with GSL will this really do much?

[bunnymustdie]

I guess we'll see how this plays out. If the core issue is with GSL will this really do much?

Probably a "trying to heal a dead horse as if it's still alive" kind of strategy. Can't really hurt, might do some small good.

[BLZrizz]

Didn't they upgrade servers a couple years ago?

I guess we'll see how this plays out. If the core issue is with GSL will this really do much?

Last upgrade was in 2015. I don’t know jack about servers but I assume hardware has improved markedly since then.

[gilchristr]

"I assume hardware has improved markedly since then"

All new servers are tier 5 with max unlocks

[Taernath]

Last upgrade was in 2015. I don’t know jack about servers but I assume hardware has improved markedly since then.

It improved markedly over the 26 years between 1989 and 2015, too. I don't think hardware is the bottleneck though, at least as far as server upgrades are concerned.

[drumpel]

The few whales this past DR probably paid Simu more than enough for new servers. So.... thank you whales???....I guess you're good to keep around for something.

[Donquix]

At least they made an announcement on the invoker. I think the lack of an announcement is what was driving most people crazy.

I think they could have also avoided a lot of ill will if they just came right out and said all of this from the very beginning: There are many issues with the game, we are trying to improve those issues, and we will monitor the changes we made to see how things work out and see if there is still room for improvement.

Simutronics seems to have a long history of getting it wrong the first time but eventually getting it right.

Literally what they've said. The entire time. Like almost verbatim, but yeah sure i guess you showed them.

[BLZrizz]

It improved markedly over the 26 years between 1989 and 2015, too. I don't think hardware is the bottleneck though, at least as far as server upgrades are concerned.

For less than the price of rift boots they can pay a small team of Indian Comp Sci PhDs to rewrite and optimize the entire code, probably.

[Tgo01]

Literally what they've said. The entire time. Like almost verbatim, but yeah sure i guess you showed them.

Your gaslighting has no power here.

[Tgo01]

Looks like the invoker shows up every few hours, announces she will hand out spells in 15 minutes, then hands out spells for 15 minutes, then goes away.

Looks like I'm a trend setter. Don't be jelly.

[Methais]

Looks like the invoker shows up every few hours, announces she will hand out spells in 15 minutes, then hands out spells for 15 minutes, then goes away.

Looks like I'm a trend setter. Don't be jelly.

That's hilarious if so.

Dreavenings in the Rift - coming soon?

[Velfi]

I don't really understand the reasoning behind making the invoker limited. If you don't want to give out every normal spell to people when they want it, then just don't have the invoker. It makes sense that Dreavenings happen every 4 hours and not on-demand, because Dreaven is an actual person, unlike the invoker.

Simu is the fucking weirdest.

[Velfi]

It will come out later that the invoker was always staffed by an intern, and rather than make it an automated service (those are bad!), they chose to give their intern breaks while not discussing the true reasoning.

[Velfi]

Ever see a video of someone from say deep, rural Finland being tossed into an American city and forced to cope with the slang and culture, and just being completely befuddled? That's Simu employees trying to connect with players.

[Gizmo]

Looks like the invoker shows up every few hours, announces she will hand out spells in 15 minutes, then hands out spells for 15 minutes, then goes away.

Looks like I'm a trend setter. Don't be jelly.

Nice, even more people to steal from in the park!

Side note: Have they brought it to other towns or is it still just the Landing?

[Velfi]

Thinking more about this, the problem with Dreavenings was that too many people were gathered together in one room. Now, the invoker is gonna work only every 4 hours, so clearly these rooms will be empt-... wait a minute....

[Tgo01]

Dreavenings in the Rift - coming soon?

Darn right!

[Stanley Burrell]

Dude, I was put into 20-second RT because when SIMUCOIN DELIVER'ing + receiving Store items was (according to the GMs) creating an inordinate amount of screamers. I don't want to upset the powers that be, so I've been doing CONSIDER {self} twice, between each delivery, lol. I don't want to give them a headache, but maybe this was one of the things, discussed, that will hopefully be fine-tuned. Alright.

[Gelston]

Dude, I was put into 20-second RT because when SIMUCOIN DELIVER'ing + receiving Store items was (according to the GMs) creating an inordinate amount of screamers. I don't want to upset the powers that be, so I've been doing CONSIDER {self} twice, between each delivery, lol. I don't want to give them a headache, but maybe this was one of the things, discussed, that will hopefully be fine-tuned. Alright.

As a former member of staff, set off all the fucking screamers you can. They need to fix that shit.

[bunnymustdie]

As a former member of staff, set off all the fucking screamers you can. They need to fix that shit.

What's a screamer? Like a notification of some kind?

[Methais]

Thinking more about this, the problem with Dreavenings was that too many people were gathered together in one room. Now, the invoker is gonna work only every 4 hours, so clearly these rooms will be empt-... wait a minute....

I'm pretty sure the biggest difference, other than people popping in, getting spells, and then leaving, is that when you get spelled up, only you see it instead of everyone in the entire room seeing every spell that's being cast on everybody.

[AnOrdim]

The picture of the new hardware confirms that if anyone ever gets the files for the servers it could literally be run from your garage with minimal difference. That is some cheap ass hardware being put in while prices at DR keep getting crazier and crazier. Stillfront getting their monies worth out of simutronics thats for sure.

[BLZrizz]

The picture of the new hardware confirms that if anyone ever gets the files for the servers it could literally be run from your garage with minimal difference. That is some cheap ass hardware being put in while prices at DR keep getting crazier and crazier. Stillfront getting their monies worth out of simutronics thats for sure.

I'm interested in the technical aspects of this, can you explain what we're seeing and why it's cheap?

[AnOrdim]

I'm interested in the technical aspects of this, can you explain what we're seeing and why it's cheap?

What you see is a "white box" server, basically a build your own. For a business, this is usually done because the perception is you are saving money. There was also specific parts mentioned in discord, the processors are workstation-class XEONs, which are fine on paper because they have very fast clock speeds which is what will help a single-threaded process the most and some 860 SSDs which are Samsung's consumer class drives. On paper all of these parts are perfectly fine and generally speaking should perform just fine. Depending on how many SSDs they decided to RAID together and in what format I'd question if the onboard RAID controller of the unknown motherboard could properly handle the bandwidth of multiple SSDs but I'm also not sure if storage bandwidth matters as much for the game or not.

Building your own server, much like someone building their own computer, leaves you to deal with each part manufacturer separately if something breaks. In a home environment this is usually fine because you are accepting your own personal level of fault tolerance and downtime. In a business, you are impacting your bottom line and your customers. The parts described could easily be replicated in a very cost-effective build from HP or Dell with a matching 3 year + 2 hour response time onsite warranty, meaning if something breaks they will usually be able to get a replacement part onsite the same day and get things going again. I also don't see, but this doesn't mean it doesn't exist on the box, any out of band support access such as ILO or IDRAC that allows someone to remotely manage a server even if turned off.

It's basically gambling the downtime over the next X years of service of the devices. Maybe they stocked up a few extra spare parts and have an IT guy that sits there all day anyway that can quickly repair them. Maybe they just don't care and want to make the absolute highest profit margin. It could even be something inbetween. Without knowing the exact model number of things like the motherboard and RAM it starts to get all theory crafting etc, but for me the choice between a whitebox server over a major manufacturer's build can only point to razor-thin budgets.

[Gelston]

What's a screamer? Like a notification of some kind?

Yes. It spams all the GM's screens in all caps.

[AnOrdim]

I think I found the motherboard:
https://www.dihuni.com/product/supermicro-mbd-x11srl-f-o-x11srl-f-server-motherboard-intel-chipset-socket-r4-lga-2066/

https://www.supermicro.com/en/products/motherboard/X11SRL-F

[bunnymustdie]

Yes. It spams all the GM's screens in all caps.

That sounds like a bizarre feature. With several hundred active players online, surely there would frequently be different people redeeming stuff from the store, some doing multiple items at once. What good is the screamer feature meant to serve?

[Gelston]

That sounds like a bizarre feature. With several hundred active players online, surely there would frequently be different people redeeming stuff from the store, some doing multiple items at once. What good is the screamer feature meant to serve?

It lets GMs know something isn't working how it is supposed to. It isn't a super common occurrence, or at least, not when I was there.

[Whirlin]

Building your own server, much like someone building their own computer, leaves you to deal with each part manufacturer separately if something breaks. In a home environment this is usually fine because you are accepting your own personal level of fault tolerance and downtime. In a business, you are impacting your bottom line and your customers. The parts described could easily be replicated in a very cost-effective build from HP or Dell with a matching 3 year + 2 hour response time onsite warranty, meaning if something breaks they will usually be able to get a replacement part onsite the same day and get things going again. I also don't see, but this doesn't mean it doesn't exist on the box, any out of band support access such as ILO or IDRAC that allows someone to remotely manage a server even if turned off.

Everything you stated is absolutely true.

However, I think you're downplaying the costs of the hardware and support, overstating the benefits. The conversion of a one-time build charge to not only have a more expensive initial build through a particular party, but then adding in a continued maintenance cost. The next-best-alternative to the hardware contracts by having a GM within range to provide onsite support, which they already have on their payroll. With the comparative costs, they could afford having a few additional components on hand and configured to swap in case of a particular drive outage. Once again, you're right, this is cheaper on two fronts... and I realize the margins are thinner now than they used to be 5 years ago, but boutique computers, especially in the enterprise space, can run at 2x on the initial build, with 5-10% additional per month depending on service provider on an ongoing basis. The difference in costs could be used to reserve a few additional components for on-site support. Additionally... support isn't always supporting and helpful.... I'll just leave it at that.

Secondly, your statements also imply there's no redundancy in either the physical or logical layers, and any individual component crash results in an outage or loss. I'd be surprised if this is the case with seemingly identical architectures.

Ideally though, your assertions are 100% spot on... however, I think in this particular use case, their position is defendable.

[Orthin]

Everything you stated is absolutely true.

However, I think you're downplaying the costs of the hardware and support, overstating the benefits. The conversion of a one-time build charge to not only have a more expensive initial build through a particular party, but then adding in a continued maintenance cost. The next-best-alternative to the hardware contracts by having a GM within range to provide onsite support, which they already have on their payroll. With the comparative costs, they could afford having a few additional components on hand and configured to swap in case of a particular drive outage. Once again, you're right, this is cheaper on two fronts... and I realize the margins are thinner now than they used to be 5 years ago, but boutique computers, especially in the enterprise space, can run at 2x on the initial build, with 5-10% additional per month depending on service provider on an ongoing basis. The difference in costs could be used to reserve a few additional components for on-site support. Additionally... support isn't always supporting and helpful.... I'll just leave it at that.

Secondly, your statements also imply there's no redundancy in either the physical or logical layers, and any individual component crash results in an outage or loss. I'd be surprised if this is the case with seemingly identical architectures.

Ideally though, your assertions are 100% spot on... however, I think in this particular use case, their position is defendable.

Well thought out and meaningful responses have no place round these parts feller’ this here’s the PC!

[Berost]

Well thought out and meaningful responses have no place round these parts feller’ this here’s the PC!

:flamewar:

[Shaps]

:flamewar:

Berost I love your profile pic. Now that is a Dwarf. /salute

[AnOrdim]

Everything you stated is absolutely true.

However, I think you're downplaying the costs of the hardware and support, overstating the benefits. The conversion of a one-time build charge to not only have a more expensive initial build through a particular party, but then adding in a continued maintenance cost. The next-best-alternative to the hardware contracts by having a GM within range to provide onsite support, which they already have on their payroll. With the comparative costs, they could afford having a few additional components on hand and configured to swap in case of a particular drive outage. Once again, you're right, this is cheaper on two fronts... and I realize the margins are thinner now than they used to be 5 years ago, but boutique computers, especially in the enterprise space, can run at 2x on the initial build, with 5-10% additional per month depending on service provider on an ongoing basis. The difference in costs could be used to reserve a few additional components for on-site support. Additionally... support isn't always supporting and helpful.... I'll just leave it at that.

Secondly, your statements also imply there's no redundancy in either the physical or logical layers, and any individual component crash results in an outage or loss. I'd be surprised if this is the case with seemingly identical architectures.

Ideally though, your assertions are 100% spot on... however, I think in this particular use case, their position is defendable.

I went through the portals I have to spec stuff out and was having a hard time even finding a true rack mount server with that particular processor. It is mostly available in workstation configurations as the name implies. So I'll give them the benefit of the doubt and say they did their homework and determined that the best-case scenario for performance was the highest clock speed processor they could get within a certain price range. Compared to the Bronze/silver/gold/platinum level Xeon "scalable" server processors, the W-2245 has a much higher core and boost speed at a fraction of the cost. There certainly are a lot of unknowns, but let's say they were to employ a clustered environment. I think that just reinforces my point, particularly with the consumer-grade SSDs. The controllers are not designed for proper RAID features and while yes they will probably work, that doesn't mean you should be doing it that way. That's not even getting into if they actually bought a proper RAID controller card or are just going to use the software RAID controller that comes with the board. I'm not seeing a capable RAID controller on that board anywhere. We also have little to no information, that I have access to anyway, on what OS they will be running etc to know if clustering is possible to provide any redundancy.

To be clear, I think the hardware is fine on paper and would be a perfectly serviceable garage server or testing environment for a business. But I would never put something like that into production let alone an internet-facing service that generates revenue. Even the budget parts total can be argued to not be "cheap" but when you are charging a subscription with multiple tiers on top of frequent direct-purchase events I would have expected a little more invested into the foundation of the house. This is all coming from the same company that has consistently been unable to replace their SSL cert in a timely fashion before it expires.


You stupid piece of shit.

[Taernath]

You stupid piece of shit.

https://media.giphy.com/media/ToMjGpjpXMFPshSYGLm/giphy.gif

[AnOrdim]

https://media.giphy.com/media/ToMjGpjpXMFPshSYGLm/giphy.gif

The peanut gallery was getting restless so I tossed them a bone.

[Methais]

Well thought out and meaningful responses have no place round these parts feller’ this here’s the PC!

QUICK, CALL HIM A DICK OR SOMETHING BEFORE IT'S TOO LATE!!!!

EDIT:


You stupid piece of shit.

Nevermind. :lol:

[Stanley Burrell]

Dude, I was put into 20-second RT because when SIMUCOIN DELIVER'ing + receiving Store items was (according to the GMs) creating an inordinate amount of screamers. I don't want to upset the powers that be, so I've been doing CONSIDER {self} twice, between each delivery, lol. I don't want to give them a headache, but maybe this was one of the things, discussed, that will hopefully be fine-tuned. Alright.
As a former member of staff, set off all the fucking screamers you can. They need to fix that shit.

Ahhh, not so sure I want to set off all the screamers (I have a few that SORT with my buggy Skellie absolutely makes miserable for the higher-ups.)

And of course, the time I was trying to swim into Hisskra's and while under water, swimming was causing screamers because I had a Paladin-bonded weapon on me and I was like, "lol. Time to drown."

I would be curious to *know* which ones are still there, but I don't want to be thrown into the abyss by the GameMasters. And then be eaten by a Grue. That's how that happens I think.



Also, I fucked up the quote tag placement in this post and am too lazy to fix it. Bye.

[mgoddess]

I would be curious to *know* which ones are still there

I've heard the morphing runestaves (that can morph into wristlets, belts, etc.) are broken enough to send screamers out... I'm tempted to try, but don't want the GM-slapdown. heh

[Mr. Dallas]

My favorite was swinging a true black ora and an Iasha white ora as a dual wield as a goof which resulted in 0 RT and the black ora being able to hit undead . . .I was mowing things down as quickly as I could mash they keys - LOL.

GM comes down and goes "do you know how many alarms that is setting off - what are you doing?" It was fun while it lasted! :)




I've heard the morphing runestaves (that can morph into wristlets, belts, etc.) are broken enough to send screamers out... I'm tempted to try, but don't want the GM-slapdown. heh

[Methais]

I've heard the morphing runestaves (that can morph into wristlets, belts, etc.) are broken enough to send screamers out... I'm tempted to try, but don't want the GM-slapdown. heh

"How dare you alert us to this stuff that needs to be fixed that we keep not fixing!" -Simu, probably

[caelric]

"How dare you alert us to this stuff that needs to be fixed that we keep not fixing!" -Simu, probably

Given that one dude on the forums got chewed out by Aulis for stating he was going keep posting until his problem item was fixed (after it having been broken for at least a year...), that is definitely the Simu approach.

[Methais]

Given that one dude on the forums got chewed out by Aulis for stating he was going keep posting until his problem item was fixed (after it having been broken for at least a year...), that is definitely the Simu approach.

Aulis is a gigantic cunt 24/7 too, so there's also that.

[caelric]

Fair point. And true.

[audioserf]

Given that one dude on the forums got chewed out by Aulis for stating he was going keep posting until his problem item was fixed (after it having been broken for at least a year...), that is definitely the Simu approach.

That person is like weaponized autism and is machine gun shitposting, fuck them imo. Even among bad posters (the officials is full of them) they stand out.

[caelric]

Maybe so. But Aulis's response is typical of how they treat anyone who is not a complete sycophant. Or Krakii. And sometimes Doug.

[Methais]

That person is like weaponized autism and is machine gun shitposting, fuck them imo. Even among bad posters (the officials is full of them) they stand out.

I'd probably be pissed too if I spent cash on text items that stayed broken for months because the GM never bothered to fix it, or whatever his deal was.

[bunnymustdie]

I'd probably be pissed too if I spent cash on text items that stayed broken for months because the GM never bothered to fix it, or whatever his deal was.

Yeah I'd be pissed too in that scenario. People spend too much on this game, that then sets them up to be disappointed when their higher expectations are unmet. If you switch to keeping only 1 basic account with just a couple of characters on it, you don't get as pissed if they mess up a little since it's only a $15-$20 dollar a month subscription game that the employees are screwing up on.

Not to make fun of the big spenders or anything, I've been guilty of the same myself both here and in other games. When you make polite feedback on stuff you think is wrong and nothing gets done or you get a "my way or the high way" vibe from the devs, the only option you got left as a customer is to vote with your wallet. Complaining more or trying to somehow negotiate with the company in question ain't gonna work.

[Methais]

Yeah I'd be pissed too in that scenario. People spend too much on this game, that then sets them up to be disappointed when their higher expectations are unmet. If you switch to keeping only 1 basic account with just a couple of characters on it, you don't get as pissed if they mess up a little since it's only a $15-$20 dollar a month subscription game that the employees are screwing up on.

Not to make fun of the big spenders or anything, I've been guilty of the same myself both here and in other games. When you make polite feedback on stuff you think is wrong and nothing gets done or you get a "my way or the high way" vibe from the devs, the only option you got left as a customer is to vote with your wallet. Complaining more or trying to somehow negotiate with the company in question ain't gonna work.

The problem with that is you can't vote with your wallet after they already have your money, and they'll never know or even notice that you didn't spend money on something else later on.

[Dud]

Is it a big surprise to anybody that the B-team runs this game?

Just because you pay a subscription fee doesn't mean they have their shit together.