Hey, so I've been watching this on and off, and I'd like to clarify a few things.

1. The big post covering the timing issue is at http://aaronin.jp/boards/viewtopic.php?p=373951#373951

2. Switching OSes will make a big difference. PIUIO is USB 2.0-capable as hell (gets 300-400 Hz on my Debian drive), but the RoXoR drive is not. The JWA fix is intended for RoXoR drives; if you're not using a RoXoR drive, you should set it to 0 in Static.ini, in the interest of fairness.

3. The Andamiro I/O controller got 60-70 Hz in our speed tests, whereas the RoXoR I/O controller got an unplayably low 30-40 Hz. I'm not sure why, but it seems to be related to the overhead of context switching. The kernel hack does all 8 necessary I/O operations in kernel space, and both controllers seem to get 100-110 Hz with that.

4. If you have any other questions about stuff, please ask me directly (vyhdycokio on AIM, or I'll occasionally poke around here). I can answer pretty much all this stuff; it's a niche specialty.

EDIT:



Correction: for a long time, I thought that was the official value too, and OpenITG originally forced it to 0 because of that. We kept getting complaints about it feeling offsync (and tighter on dedicabs specifically), and a lot of research later, we discovered that as one of the causes. (The other one was the I/O thread priority, which has been fixed in beta 3, but that's more of an edge case.) We got a lot of FUD about "being offsync" for it, so that's entirely my mistake for not objectively testing it in the first place.

I'm going tonight probably around 7.

Can you simplify this in lamen terms? Sorry, but my puny brain can't fully grasp what you just wrote.

The parts about OpenITG might be slightly off, since I don't know the code at all and lower level systems stuff is not my specialty, but this should be a thorough pass over everything:

Context switching is a key point when it comes to a computer running multiple programs at once.  Ignoring multi-core processors (for simplicity - I'll come back to them at the end), "multitasking" is actually the computer switching out the currently running thread (a collection of instructions and related data - any visible program usually has multiple threads, but that's a bit of a tangent) for one that's been on hold; this is done hundreds of times per second when a computer is under load.  A processor works in sequence, plain and simple, but a computer also works so fast that if it spends a little time on each active program, a human won't perceive the downtime (unless a lot of things are running at once).

A context switch requires dumping the data currently in the processor's registers out to RAM, then calling up the data and instructions of the new thread; the processor sits and waits for this to finish, since you can't do anything with a half-loaded thread.  In terms of CPU cycles, this takes a decent chunk of time (it can take multiple microseconds on modern server hardware), so in a case like the OpenITG I/O code, where multiple kernel switches are needed in an otherwise short sequence of operations, context switches make up an overwhelming majority of the execution time.  Context switching is slower when the next thread is from outside the current thread's process (that is, the processor is switching to a second program, which could be the OS or something else you have running), since multiple threads in the same process share a decent amount of data, and kernel code is definitely outside the current process, so things are compounded a bit.

The core parts of the OS are called the kernel.  Kernel code has special protections enforced on it, and you must make a special call to the OS to get it to use that kernel code for you, so you have expensive context switches (into, then out of the protected kernel mode) as well as some extra cost from making a special protected call instead of accessing the kernel code directly.  The lowest levels of I/O logic are usually protected kernel code, though not always and those parts can be done without switching to kernel mode.  Originally, RoXoR's code did the safe (least buggy, generally fairly efficient) thing and had their code make system calls only when necessary, doing as much in user mode as possible, but this introduced a lot of context switching, since the code waits for the system call to finish before continuing.  Their hacked I/O system lumped every bit of I/O work into a single kernel mode call that dumped a bunch of results all at once, which is a bit messier and bug prone, but this gave significant performance gains, since multiple context switches we're collapsed into a single context switch.

As for modern multi-core and/or hyperthreaded processors, this all still relevant.  A quad-core processor runs four threads at once, but a fresh install of any OS will have many more than two programs running at startup, and many (if not all) of them will be multithreaded, so do have some degree of real multitasking across processors, but you'll still be switching things out constantly.  Intel's Hyperthreading concept was a fairly effective attempt to address the cost of context switching by giving a processor two full sets of registers, allowing one set to do actual work while the other moves data around.  In either case, the user-kernel switching described is a lot less painful, since other things can happen while the context switching is happening.

The Celeron RoXoR used is a single core processor that also lacks Hyperthreading, so those context switches are very expensive (actually, even current dual-core Celerons still skip out on Hyperthreading to bring the price tag down a bit).  Ultimately, a major bottleneck was USB 1.1, which is why running OpenITG on a USB 2.0 (which has about 12 times the maximum transfer rate of 1.1, IIRC) friendly OS gets a huge performance gain.  A processor which handles context switching faster probably gets performance gains as well, but the returns are diminishing because of all the other stuff that the computer is doing (the game spins up quite a few threads on its own, plus the OS runs a bunch of background programs), which the OS tries to schedule as fairly as possible.  I would be curious to see what kind of polling rate the new Acme computer can get - I'll probably track down vyhd on AIM and find out how to benchmark the polling rate.

When I asked for it in lamen terms, I meant that I needed it simplified. An analogy, if you will.

From what I gather:
Original ITG gave priority to the pad polling. OpenITG doesn't give priority to any one specific task. Is this more-or-less correct?

LOL at Tony's "simplification".

@ Chris:  Is DM's drum better?  It was pulled apart and had a run through.  But not much was evident as far as a problem.  Is it fast repetition?

@ Allan, how is that arrow today?

Not quite.  Basically, some of the polling code depends on protected kernel "tools", the rest does not.  Rather than crossing a security checkpoint to use a kernel tool and crossing back once done with it (and going through the checkpoint again for a few other tools), they moved all their tools into the secure area and go into it once, do everything, then leave.

There was a separate point about raising the I/O thread priority over other OpenITG threads in Beta 3, which I remember reading was meant more to address load screens stealing coins than to address the polling rate - as vyhd said, it's an edge case for thread priority to help the polling rate.  The loading thread may or may not have had higher priority than the I/O thread (vyhd would know), but whatever the case, the relative priorities kept the scheduler from letting the I/O thread run at all while data was loading. which would cause multi-second periods with no I/O polling.

And yeah, the full bottom up description was hardly succinct, but I couldn't tell if the request was for a simplification or a comparably detailed explanation that either skipped or explained jargon as appropriate.

Computer systems are abstraction upon abstraction, so things really blow up when you try to cover something that runs from low-level I/O handling up to interactions between the OS and specific programs - in 99% of cases, 99% of that description is stuff that just kind of works because the OS, drivers, etc., were all written well enough that you don't run into major issues anywhere, and you therefore don't even have to think about it - case in point, upgrade cabs run very smoothly at 160 Hz without having to do detailed kernel level work, but the switch from JAMMA to USB for a dedicab introduced a subtle low level problem that took  until r16 to track down (as evidenced by early patches having JWA=0.0015 as a triage hack).

Was working great yesterday! Had a 10000 combo, broke it on purpose on hard stuff. Best I can figure it's fixed now Thanks!

Bill I haven't been by since I made the previous post.  Sorry, I will try to get to ACME this week and let you know.

Thanks for looking into it; I'm the only one that seems to have noticed any issue.  So maybe it's my own imagination.

According to ACME's website, they are now open until midnight on weekdays instead of 11!! This is awesome news for me, but I don't know if anyone else cares.

I didn't know they officially changed the time... I do know they've been open past 11 for about a month now though. It is awesome!

I got stuff about the pads, I'll post it or call it in when I have more time

That is awesome.  I might go tonight just because of learning this.

If you do, I will see you there!

Went for the first time in a long time yesterday. I'll be coming quite frequently now, as I've just had it with Gameworks. I feel bad for ever leaving -_-. Anyway, Drummania seemed pretty good to me yesterday. Hihat and cymbal feel "loose" as usual, but everything works, which is more than I can say about Gameworks.

Going probably around 9 tonight.

Has anyone else been getting way offs on the 2P side? I've been getting a ton, and I guess it must just be due to my bad playstyle or something. Anyway, the machine was stuck in the BIOS with a "please remove media" message (or something similar) but there wasn't a flash drive in the machine, so I just rebooted it and it was fine. I also noticed that the 1P side shakes like something's loose or not touching the ground. Sensitivity seemed fine though.

I've had more problems on the 1st player side than 2nd player side-- I get little to no errors on the 2nd side for some reason. Yeah, the first player side really shakes a lot and I'm guessing it's due to the thingy not touching the ground making it really noisy every time someone plays on it; I feel like I'm gonna break the pads playing on that side haha.

That's my input anyway.

Pad way offs?  Now you're just making stuff up.

Any Way Off that I've gotten on ITG at Acme I feel is entirely my fault.  The only times I can remember it happening are during medium-speed or faster (i.e. 130+ bpm) 1/16 runs; any way offs I get are almost certainly a consequence of me playing no bar and not moving my feet off an arrow.

Maybe someone who plays with the bar should chime in.

Hardly.  I've noticed them myself, but only on the left side.

I cannot speak to personal experience for any errors on the left side, though I've heard of such misses and have suspected some having watched people. I want to say it's the up arrow on the left side, but we had a tech check that one out and clean it, so I'm not sure.

As for the right side, I have definitely had a couple of errors in the last couple days; I am almost positive it was the right side and I think the arrow was just sticking a little. I lost a roll on the right side and a couple way offs in various streams so I think it's worth taking a look. It's not *really* bad, but it can definitely be annoying occasionally and could feasibly degrade from here.

Anyway thanks for looking into it Bill!

Pad Way Offs are usually echos, rather than weak sensors.  Weak sensors will either mean it takes a tiny bit longer to register when you step lightly (giving you Excellents or Greats, normally - really broken Extremes have given me pad Goods, but Acme's ITG has never gone long enough without maintenance to get to pad Decents) or that a light step doesn't register at all.  Given ITG's polling rate causing the timing windows to shift slightly for every step, detecting errors from weak sensors (other than misses) can be extremely difficult.

A normal (or over-sensitive, due to too much tension in the bracket that sits between the sensor and arrow) sensor can sometimes trigger because the arrow is bouncing after you step hard, then pull your weight back off quickly - this would give you a Way Off.  ITG has a "debounce" setting that's supposed to detect and ignore such noise fairly well and the panel sits with less wiggle room that on a normal DDR machine, so echoes are unlikely, but possible - I can't comment, since I almost never get echoes on any machine, since I only step hard in sections that are slow enough that I'll keep my weight on the arrow for awhile.  Echoes are definitely something that is tied to technique and the player's build.  Another possible explanation (this does happen to me) - if you tap out extra steps to keep your rhythm, you might be stepping closely enough to the next arrow on that panel as to trigger it - stepping half a beat early on a song that's over ~160 bpm will almost always net you a Way Off.

Fixing an overly sensitive arrow is not fun.  I've had it happen at Sakuracon twice now; I clean the gunk off a sensor that's been giving misses, put it pack in place, and find that it will now trigger if I drop two coins on it from a foot above the arrow (or that the weight of the arrow alone is enough to trigger it).  Usually I hunt around for a slightly weaker outer sensor somewhere and swap it, so that the bracket tension compensates for the weaker sensor and the strong sensor has less tension on it and is now sitting in a less critical location.  I don't know if Bill's techs normally check that closely for a strong sensor when they look at the machine, but it's worth looking into if people are getting Way Off.

There's a lot of nuance things about maintaining these games that would be impossible for the average technician to know to look for, unless they are serious into DDR themselves.

The pad Way Offs might also be caused by an exposed wire on the sensors rubbing against metal. This is even more likely if a player is "shuffling" on the arrow, rather than depressing it straight down. I'd suggest making sure none of the sensor's connector wires are exposed anywhere.

Tom and I were experiencing that with his machine on the right side.  It was giving lots of Way Offs on different arrows, even though the issue was just with one arrow.  It turned out to be an over-sensitive sensor (hurr hurr) along with another sensor with its wires exposed.  The way that we find over-sensitive stuff is just by stomping on the metal next to the sensors.  If it lights up, it's probably over-sensitive.  I've had great success finding them that way, and it's a fairly easy test. =p

EDIT:  Also, I've found the only cure for an over-sensitive sensor is to just replace it.  There's no good fix for it.  So like Tony says, if there are no new sensors available, I would swap it for an outer sensor that's in better condition.

@Tricksy,
Nice analogy!  Actually everyone here should be arrow smashing game techs.  But yes, when a sensor is too sensitive, it belongs in the trash can!  Does Tony or anyone have sensors left over from tournaments in the past?  If so, and you are there, and you spot a bad one, feel free to swap it out.  :-)