NES cart dumping/plug-in

[nitro322]

I know this question comes up a lot, but rather than talk about how awesome it would be to have a plugin for this (though I won't lie - it would be quite awesome), I'd like to have a discussion about why it's so difficult to create one.  Specifically: what are the challenges involved in dumping NES carts, and are there any practical solutions for overcoming or working around them?

As I understand it, based on rather limited research, here are the fundamental challenges of dumping NES carts:

• Each cart has (at least?) two types of ROM chips: PRG+CHR
• Each cart may use one of many different, incompatible types of memory mappers
• There is no programmatic way from the ROM(s) themselves to determine which memory mappers, etc. are included or should be used (ie., no built-in header)

Is that at least somewhat accurate?  I'm sure I'm missing and/or oversimplifying some details, so please correct me where appropriate.

As for possible solutions... I know that there are a lot of people working on documenting NES ROMs.  One of the most thorough and complete databases that I've found is the quite aptly named NES Cart Database.  The even publish an XML document to make the data available for programmatic use (this will be referred to as 'database' below).

So, is there any reason Retrode, or an appropriate NES Retrode plugin, couldn't leverage this data to workaround the limitations of the NES carts themselves?  For example, I'm thinking of something like this:

• A copy of the database is either baked into the firmware or exposed separately on the device (similar to RETRODE.CFG) for easier updates
• When reading a NES cart, a checksum is performed of the PRG and CHR ROMs
• The checksums are compared against the database to determine which game is being read
• Using the information obtained from the database, the ROM can be dumped in one of two different ways:
  
  • Separate ROMs for the PRG and CHR, plus an extra XML file (or whatever) to define the data
  • A combined data file w/ an appropriate header to define the data (such as iNES or UNIF format)

I think 4a would be better/cleaner, but 4b would be acceptable as well.  In either case, the result would be a successful dump of the NES cart through a Retrode, as joy and happiness abound.

Again, I'm sure I'm missing some details or (grossly) oversimplifying the technical challenges, but conceptually, is something like this feasible?  If not, why not, and are there any other feasible (if not easy) solutions available?  I'd like to better understand the problems/challenges involved.

[emuDrache]

As I understand it (from the various posts Matthias has made on the subject),
you hit the nail right on the head.

Too many mappers, and no way to auto-detect which one is used.
The Retrode would have to be configured on a per cart basis.

re #1 - Matthias has said that there is no room left in the firmware for something like that.

having said that, maybe Matthias could do a custom adapter with jumpers on it or something,
on the 16 bit port, but I seriously doubt it would work cleanly, and would probably require an external program to clean up the dump afterwards.

Overall, it would probably require a custom firmware dedicated to NES carts to do that.

Which wouldn't much bother me as I have 2 original Retrode's, and 1 Retrode 2, but not everyone
is in that situation.

The database side would be simple to do.  I was considering doing one for the Atari 2600 to
'fix' the ROMs as dumped by the Retrode, since a few are not right due to overdump/swapped bytes
(i don't remember the specifics at the moment) caused by the primitive "mappers" used by the Atari 2600 on a few carts.

Regards,
Felix

So, is there any reason Retrode, or an appropriate NES Retrode plugin, couldn't leverage this data to workaround the limitations of the NES carts themselves?  For example, I'm thinking of something like this:

• A copy of the database is either baked into the firmware or exposed separately on the device (similar to RETRODE.CFG) for easier updates
• When reading a NES cart, a checksum is performed of the PRG and CHR ROMs
• The checksums are compared against the database to determine which game is being read
• Using the information obtained from the database, the ROM can be dumped in one of two different ways:
  
  • Separate ROMs for the PRG and CHR, plus an extra XML file (or whatever) to define the data
  • A combined data file w/ an appropriate header to define the data (such as iNES or UNIF format)

[Muzer]

Yes, there was a short discussion in the comments about this. Matthias_H said that memory is not a problem as he could always include a chip in the plug-in itself.

I was going to generate a database at some time during the next few weeks, of checksums of the first few kilobytes of data and mappers/etc.

[emuDrache]

interesting.  i never saw the post where he talked about the chip on the plugin itself.

[Muzer]

http://www.retrode.org/2012/03/plug-in-prototypes/comment-page-1/#comment-26051

[emuDrache]

sounds good

now we just need a willing developer

[Matthias_H]

Yes, there was a short discussion in the comments about this. Matthias_H said that memory is not a problem as he could always include a chip in the plug-in itself.

Not quite. I said that it could technically be done, but I did by no means say that I'm ever going to touch NES altogether. With the very limited time on my hand, I have to make choices on what aspects of the Retrode to best spend it on. The NES is definitely not one of them -- truly sorry, but I simply lack any emotional attachment to that era of video gaming. Find me enthusiastic and capable programmers, and I'll be more than happy to share my code with them.

[Muzer]

Right, sorry.

[nitro322]

So, something along the lines of what I suggested would be technically possible, but at this point we lack the interested of a capable developer/engineer?  Is that an accurate assessment?

Hey, Matthias, I can understand where you're coming from.  By contrast, the NES and SNES are intertwined in their sheer awesomeness for me, whereas I've never owned a Genesis in my life and have zero use for that port (unless, maybe, for an adapter).  So, I get you there.  Looking at it from a different perspective, though, best I can tell there are no easy-to-use*, pre-built available NES cart dumpers readily available.  A NES plugin for the Retrode would be the first of its kind, and a very strong selling point for the device, helping you move more units.

Of course, if you're doing this for the love and challenge rather than the money (which I can totally respect), that's probably not going to matter a whole lot, but I wanted to throw it out there.  Otherwise, I can only hope some other crazy awesome retrogamer heeds the call.  :-)

*As awesome as the USB CopyNES looks, requiring physical modification of, and soldering to, a real NES fails my easy-to-use test, and for someone as clumsy as me it makes me fear for the safety of my beloved NES.

[Matthias_H]

It's great to see you folks pursue these ideas with such enthusiasm, and again, I'll be happy to do my best to assist anyone who will take on the NES. It's just that I predict that it will be a massive effort with lots of frustrating moments ("ah well, another thing we're going to need a database for"). And that I won't be the one in charge and pushing the whole thing forward.

Mind you, I'm doing this on top of a regular daytime job that, taken alone, isn't exactly 9-to-5. As much as I would love to respond to the popular demand for NES and a whole pile of other stuff, I simply lack the capacity for it. After the Retrode 2 release, I also owe it to myself and, especially, my family to radically cut back on all the "extra stuff" and focus on the essentials. Sadly, there are other, way more urgent things that also need to be done (e.g., found a Limited as soon as possible to get rid of private liability).

[nitro322]

Fair enough.  I'm satisfied that it is (or seems to be) at least technically possible, which was the main thing I was wanting to get out of this thread.  I can at least hope that someone with the time and desire to work on this will be able to do so in the future.  :-)

Appreciate the responses.

[Matthias_H]

Just wondering - Have you ever considered channeling your, uhm, "detail orientation" into something that would enable you to actually get things done? Instead of having to hope for that programming messiah, you might as well teach yourself some programming, and basic soldering skills aren't that hard to obtain either. Time doesn't seem to be in short supply for you

[nitro322]

Time doesn't seem to be in short supply for you

Hmm...  I'll try not to take too much offense from that.  :-)

I've thought about it.  Definitely want to do the soldering thing at some point, as there are a bunch of projects along the lines of CopyNES that I'd like to do one day.  But, as with so many other things I just never get around to it.  Too much other stuff constantly vying for my attention, and oh so many distractions and little side projects to work on (such as randomly deciding to dump all of my old SNES games for no other reason than "because I could...").

Just to be clear, I'm not exactly sitting here waiting for a 'programming messiah' to solve all my ills or provide all my desires (though I wouldn't exactly turn that down, either ;-) ).  I can program just fine as it is, but low-level stuff has never particularly interested me, nor has it ever "clicked" with me the times I've tried.  I prefer to stick to higher level scripting languages - I'm much more productive with them, and I find them more enjoyable to work with.  If I could write the NES plugin firmware in bash or PHP or something, I'd be all over it.  As it is, I'll let the hardware hackers handle the stuff they're good at, and I'll stick with the stuff I tend to enjoy.

In this particular case, I do think it'd be nice if someone with the interest and desire decided to take this on; I'd love to dump all of my NES games as well, and again, just because I could.  If it doesn't happen, though, no big deal.  I already got what I initially wanted and expected out of my Retrode, and I'll be quite content with that if that's all I get.  Nonetheless, I'll still hold out some hope that I may be able to utilize it for even more in the future.

[Matthias_H]

Hmm...  I'll try not to take too much offense from that.  :-)

Good choice, because I tend to be randomly offensive whenever I feel like it Don't really mean it, though. I'm generally a nice guy, if admittedly a bit stressed out atm.

If I could write the NES plugin firmware in bash or PHP or something, I'd be all over it.  As it is, I'll let the hardware hackers handle the stuff they're good at, and I'll stick with the stuff I tend to enjoy.

You'd actually prefer bash over C? That's just about the oddest thing I heard today, and we had quite some odd stuff going on earlier. PHP, not so different except there you don't really need to care about types and memory allocation and all that.

In this particular case, I do think it'd be nice if someone with the interest and desire decided to take this on; I'd love to dump all of my NES games as well, and again, just because I could.  If it doesn't happen, though, no big deal.

Well, thing is, in my experience the likelihood of that someone showing up out of nowhere is near zero. With around 1k Retrode users out there, we're slowly reaching a critical mass where questions and problems on the forum start being solved without me having to constantly kick in. It will probably take quite a few more folks until actual development issues can be dealt with in the same way. Meanwhile, you might as well get your hands on a microcontroller and experience the good vibes that emanate from your first blinking LED (the equivalent of "Hello World") first hand

[nitro322]

You'd actually prefer bash over C? That's just about the oddest thing I heard today, and we had quite some odd stuff going on earlier.

Heh, yeah, guess I'm just odd like that.  Took me a while to get the hang of bash/shell syntax, but it's a pretty capable language in its own right, and ridiculously portable across UNIX variants.  Plus, I love being able to define arbitrarily complex aliases and functions directly within my shell.  I'm up to a 1123 line ~/.bashrc file full of useful little functions and shortcuts that I can easily copy around to whatever computer I'm currently using (hmm... maybe I do have too much free time...).  C, C++, Java [shudders], and other 'compiled' languages have just never done it for me, nor has stuff like assembly.

Well, thing is, in my experience the likelihood of that someone showing up out of nowhere is near zero.... Meanwhile, you might as well get your hands on a microcontroller and experience the good vibes that emanate from your first blinking LED (the equivalent of "Hello World") first hand

Fair enough.  Again, this is something I already would like to play around with some day, but that's not likely to be any time soon.  I have waaaay to many other projects I want to wrap up first before embarking on such a major new hobby.