I know this question comes up a lot, but rather than talk about how awesome it would be to have a plugin for this (though I won't lie - it would be quite awesome), I'd like to have a discussion about why it's so difficult to create one. Specifically: what are the challenges involved in dumping NES carts, and are there any practical solutions for overcoming or working around them?
As I understand it, based on rather limited research, here are the fundamental challenges of dumping NES carts:
- Each cart has (at least?) two types of ROM chips: PRG+CHR
- Each cart may use one of many different, incompatible types of memory mappers
- There is no programmatic way from the ROM(s) themselves to determine which memory mappers, etc. are included or should be used (ie., no built-in header)
Is that at least somewhat accurate? I'm sure I'm missing and/or oversimplifying some details, so please correct me where appropriate.
As for possible solutions... I know that there are a lot of people working on documenting NES ROMs. One of the most thorough and complete databases that I've found is the quite aptly named NES Cart Database (http://bootgod.dyndns.org:7777/). The even publish an XML document (http://bootgod.dyndns.org:7777/xml.php) to make the data available for programmatic use (this will be referred to as 'database' below).
So, is there any reason Retrode, or an appropriate NES Retrode plugin, couldn't leverage this data to workaround the limitations of the NES carts themselves? For example, I'm thinking of something like this:
- A copy of the database is either baked into the firmware or exposed separately on the device (similar to RETRODE.CFG) for easier updates
- When reading a NES cart, a checksum is performed of the PRG and CHR ROMs
- The checksums are compared against the database to determine which game is being read
- Using the information obtained from the database, the ROM can be dumped in one of two different ways:
- Separate ROMs for the PRG and CHR, plus an extra XML file (or whatever) to define the data
- A combined data file w/ an appropriate header to define the data (such as iNES or UNIF format)
I think 4a would be better/cleaner, but 4b would be acceptable as well. In either case, the result would be a successful dump of the NES cart through a Retrode, as joy and happiness abound.
Again, I'm sure I'm missing some details or (grossly) oversimplifying the technical challenges, but conceptually, is something like this feasible? If not, why not, and are there any other feasible (if not easy) solutions available? I'd like to better understand the problems/challenges involved.
As I understand it (from the various posts Matthias has made on the subject),
you hit the nail right on the head.
Too many mappers, and no way to auto-detect which one is used.
The Retrode would have to be configured on a per cart basis.
re #1 - Matthias has said that there is no room left in the firmware for something like that.
having said that, maybe Matthias could do a custom adapter with jumpers on it or something,
on the 16 bit port, but I seriously doubt it would work cleanly, and would probably require an external program to clean up the dump afterwards.
Overall, it would probably require a custom firmware dedicated to NES carts to do that.
Which wouldn't much bother me as I have 2 original Retrode's, and 1 Retrode 2, but not everyone
is in that situation.
The database side would be simple to do. I was considering doing one for the Atari 2600 to
'fix' the ROMs as dumped by the Retrode, since a few are not right due to overdump/swapped bytes
(i don't remember the specifics at the moment) caused by the primitive "mappers" used by the Atari 2600 on a few carts.
Regards,
Felix
Quote from: nitro322 on 26/Mar/2012 04:34:59 AM
So, is there any reason Retrode, or an appropriate NES Retrode plugin, couldn't leverage this data to workaround the limitations of the NES carts themselves? For example, I'm thinking of something like this:
- A copy of the database is either baked into the firmware or exposed separately on the device (similar to RETRODE.CFG) for easier updates
- When reading a NES cart, a checksum is performed of the PRG and CHR ROMs
- The checksums are compared against the database to determine which game is being read
- Using the information obtained from the database, the ROM can be dumped in one of two different ways:
- Separate ROMs for the PRG and CHR, plus an extra XML file (or whatever) to define the data
- A combined data file w/ an appropriate header to define the data (such as iNES or UNIF format)
Yes, there was a short discussion in the comments about this. Matthias_H said that memory is not a problem as he could always include a chip in the plug-in itself.
I was going to generate a database at some time during the next few weeks, of checksums of the first few kilobytes of data and mappers/etc.
interesting. i never saw the post where he talked about the chip on the plugin itself.
http://www.retrode.org/2012/03/plug-in-prototypes/comment-page-1/#comment-26051
sounds good :)
now we just need a willing developer :)
Quote from: Muzer on 26/Mar/2012 09:49:09 AM
Yes, there was a short discussion in the comments about this. Matthias_H said that memory is not a problem as he could always include a chip in the plug-in itself.
Not quite. I said that it could technically be done, but I did by no means say that I'm ever going to touch NES altogether. With the very limited time on my hand, I have to make choices on what aspects of the Retrode to best spend it on. The NES is definitely not one of them -- truly sorry, but I simply lack any emotional attachment to that era of video gaming. Find me enthusiastic and capable programmers, and I'll be more than happy to share my code with them.
Right, sorry.
So, something along the lines of what I suggested would be technically possible, but at this point we lack the interested of a capable developer/engineer? Is that an accurate assessment?
Hey, Matthias, I can understand where you're coming from. By contrast, the NES and SNES are intertwined in their sheer awesomeness for me, whereas I've never owned a Genesis in my life and have zero use for that port (unless, maybe, for an adapter). So, I get you there. Looking at it from a different perspective, though, best I can tell there are no easy-to-use*, pre-built available NES cart dumpers readily available. A NES plugin for the Retrode would be the first of its kind, and a very strong selling point for the device, helping you move more units.
Of course, if you're doing this for the love and challenge rather than the money (which I can totally respect), that's probably not going to matter a whole lot, but I wanted to throw it out there. Otherwise, I can only hope some other crazy awesome retrogamer heeds the call. :-)
*As awesome as the USB CopyNES (http://www.retrousb.com/product_info.php?products_id=36) looks, requiring physical modification of, and soldering to, a real NES fails my easy-to-use test, and for someone as clumsy as me it makes me fear for the safety of my beloved NES.
It's great to see you folks pursue these ideas with such enthusiasm, and again, I'll be happy to do my best to assist anyone who will take on the NES. It's just that I predict that it will be a massive effort with lots of frustrating moments ("ah well, another thing we're going to need a database for"). And that I won't be the one in charge and pushing the whole thing forward.
Mind you, I'm doing this on top of a regular daytime job that, taken alone, isn't exactly 9-to-5. As much as I would love to respond to the popular demand for NES and a whole pile of other stuff, I simply lack the capacity for it. After the Retrode 2 release, I also owe it to myself and, especially, my family to radically cut back on all the "extra stuff" and focus on the essentials. Sadly, there are other, way more urgent things that also need to be done (e.g., found a Limited as soon as possible to get rid of private liability).
Fair enough. I'm satisfied that it is (or seems to be) at least technically possible, which was the main thing I was wanting to get out of this thread. I can at least hope that someone with the time and desire to work on this will be able to do so in the future. :-)
Appreciate the responses.
Just wondering - Have you ever considered channeling your, uhm, "detail orientation" into something that would enable you to actually get things done? Instead of having to hope for that programming messiah, you might as well teach yourself some programming, and basic soldering skills aren't that hard to obtain either. Time doesn't seem to be in short supply for you :)
Quote from: Matthias_H on 28/Mar/2012 01:35:23 AMTime doesn't seem to be in short supply for you :)
Hmm... I'll try not to take too much offense from that. :-)
I've thought about it. Definitely want to do the soldering thing at some point, as there are a bunch of projects along the lines of CopyNES that I'd like to do one day. But, as with so many other things I just never get around to it. Too much other stuff constantly vying for my attention, and oh so many distractions and little side projects to work on (such as randomly deciding to dump all of my old SNES games for no other reason than "because I could...").
Just to be clear, I'm not exactly sitting here waiting for a 'programming messiah' to solve all my ills or provide all my desires (though I wouldn't exactly turn that down, either ;-) ). I can program just fine as it is, but low-level stuff has never particularly interested me, nor has it ever "clicked" with me the times I've tried. I prefer to stick to higher level scripting languages - I'm much more productive with them,
and I find them more enjoyable to work with. If I could write the NES plugin firmware in bash or PHP or something, I'd be all over it. As it is, I'll let the hardware hackers handle the stuff they're good at, and I'll stick with the stuff I tend to enjoy.
In this particular case, I do think it'd be nice if someone with the interest and desire decided to take this on; I'd love to dump all of my NES games as well, and again, just because I could. If it doesn't happen, though, no big deal. I already got what I initially wanted and expected out of my Retrode, and I'll be quite content with that if that's all I get. Nonetheless, I'll still hold out some hope that I may be able to utilize it for even more in the future.
Quote from: nitro322 on 28/Mar/2012 01:54:26 AM
Hmm... I'll try not to take too much offense from that. :-)
Good choice, because I tend to be randomly offensive whenever I feel like it ;) Don't really mean it, though. I'm generally a nice guy, if admittedly a bit stressed out atm.
QuoteIf I could write the NES plugin firmware in bash or PHP or something, I'd be all over it. As it is, I'll let the hardware hackers handle the stuff they're good at, and I'll stick with the stuff I tend to enjoy.
You'd actually prefer bash over C? That's just about the oddest thing I heard today, and we had quite some odd stuff going on earlier. PHP, not so different except there you don't really need to care about types and memory allocation and all that.
QuoteIn this particular case, I do think it'd be nice if someone with the interest and desire decided to take this on; I'd love to dump all of my NES games as well, and again, just because I could. If it doesn't happen, though, no big deal.
Well, thing is, in my experience the likelihood of that someone showing up out of nowhere is near zero. With around 1k Retrode users out there, we're slowly reaching a critical mass where questions and problems on the forum start being solved without me having to constantly kick in. It will probably take quite a few more folks until actual development issues can be dealt with in the same way. Meanwhile, you might as well get your hands on a microcontroller and experience the good vibes that emanate from your first blinking LED (the equivalent of "Hello World") first hand :)
Quote from: Matthias_H on 28/Mar/2012 02:21:15 AMYou'd actually prefer bash over C? That's just about the oddest thing I heard today, and we had quite some odd stuff going on earlier.
Heh, yeah, guess I'm just odd like that. Took me a while to get the hang of bash/shell syntax, but it's a pretty capable language in its own right, and ridiculously portable across UNIX variants. Plus, I love being able to define arbitrarily complex aliases and functions directly within my shell. I'm up to a 1123 line ~/.bashrc file full of useful little functions and shortcuts that I can easily copy around to whatever computer I'm currently using (hmm... maybe I do have too much free time...). C, C++, Java [shudders], and other 'compiled' languages have just never done it for me, nor has stuff like assembly.
Quote from: Matthias_H on 28/Mar/2012 02:21:15 AMWell, thing is, in my experience the likelihood of that someone showing up out of nowhere is near zero.... Meanwhile, you might as well get your hands on a microcontroller and experience the good vibes that emanate from your first blinking LED (the equivalent of "Hello World") first hand :)
Fair enough. Again, this is something I already would like to play around with some day, but that's not likely to be any time soon. I have waaaay to many other projects I want to wrap up first before embarking on such a major new hobby.
Is the firmware written in C, or have you been coding directly in whatever assembly language it uses? If the former, that would make it a whole lot easier - I was always just assuming it was the latter, and I've no idea why I was ;).
I'm reasonably competent with C, and I have used a microcontroller (but cheating a little, an Arduino, and nothing nearly as complex as cart dumping). Perhaps I should have a go once I've learnt a bit more - I'll stick with making this DB first as it should prove a lot easier.
The source is 99.9% C (pretty much everything apart from some NOPs, that is).
Is the firmware source code available? I only recall seeing the binaries on the website.
QuotePeter C
March 20th, 2012 at 6:48 pm
Those look awesome. I can't wait to give you money for the production version.
Out of curiosity what were the technical limitations for NES plugin that you mentioned?
PC
Matthias_H
March 20th, 2012 at 6:58 pm
None of the autodetection would work. You would have to manually specify PRG ROM size, CHR ROM size, mapper chip and SRAM size for each cartridge you put in. Doesn't sound like you'd actually want to do that — and if you would, you could probably build your own plug-in too.
Ness and Sonic
March 22nd, 2012 at 5:23 am
I can see your point about NES being a pain. If anyone's wanting an idea about the games and their mappers, I came across this incomplete database: http://tuxnes.sourceforge.net/nesmapper.txt
Matthias_H
March 22nd, 2012 at 5:35 am
Yeah, I'm aware of this database. Still, I can't seem to think of a smart way to look up table entries without knowing the game title. A camera + OCR processor integrated into the plug-in (to read cartridge labels) seems like a bit of overkill.
Muzer
March 22nd, 2012 at 7:09 pm
Is there no game title in some easy-to-access place in the ROM, then? Or any other uniquely identifiable information?
Actually, just looked it up a bit more – looks like you should be able to checksum the first few KB of the NES ROMs (which, the internet implies, are read in the same way), and compare it with a list that someone (could be me) has generated. The only issue is, would such a database actually fit into the limited memory capacity of the Retrode device?
Matthias_H
March 22nd, 2012 at 7:53 pm
Memory shouldn't be an issue – the plug-in itself could contain an additional flash ROM / EEPROM chip for this purpose. Still, sounds like a horrible lot of work to me.
Muzer
March 22nd, 2012 at 8:41 pm
Generating the DB shouldn't be particularly hard, though – I think I'll give it a try some time soon. I'll just use some simple checksum on the first couple of kilobytes, hopefully enough to be unique.
andwhyisit
March 28th, 2012 at 12:36 am
I can see this creating any number of false positives. Plus a database would be annoying to maintain.
Just dump only the first bank by default, place the checksum in the filename, then allow up to 10 NES filename/PRG ROM size/CHR ROM size/mapper chip number/SRAM size configurations in the config file. You can check what the generated filename would be, then you can look up the configuration to see if the filename was recorded and then load the associated configuration for dumping the file. This preserves the plug and play aspect for up to 10 different carts.
Muzer
March 30th, 2012 at 8:02 am
Annoying to maintain? How many NES games are being released nowadays?
Your suggestion would work, but I don't see how a database would be much harder. It will be relatively-speaking trivial for me to generate one without any human input.
andwhyisit
April 3rd, 2012 at 2:15 pm
To obtain the checksums for the database you would have to dump the beginning of the game, calculate the checksums yourself, send it to Matthias, and wait for him to submit an update to the plugin's firmware, which would now be much more expensive because of the additional hardware to store the required database. It's inefficient. If you want a database then how about adding one to the site? This way retrode users can look up their NES configuation sets and can just copy them to their retrode config.
Matthias certainly isn't going to go out and buy or pirate the entire NES library for the sake of creating a database, nor will he accept a dump of the first bank of a game to calculate checksums, so it is down to users to dump the first bank and calculate the checksums, so the database will always be incomplete.
The way I suggested is more efficient, cheaper, easier to implement, flexible, more user friendly, doesn't require constant updates to some database (which may or may not be possible under the current retrode hardware) that will fail you the moment you insert some cart into the retrode that shares its checksum with more than one database entry, and will work for every single cart whose mapper is supported by the retrode.
Plus, unlicenced NES games are still released to this day if that answers your question.
Muzer
April 4th, 2012 at 11:01 am
I never at any point stated nor implied that Matthias would be the one making the initial database. In fact, I've already said that I will try to make one at some point in the near future.
Homebrew NES games have been released, true, but few on actual cartridges – and besides, those aren't exactly in the scope of what the Retrode was intended for.
andwhyisit
April 5th, 2012 at 1:00 am
I wasn't only talking about homebrew titles but also the stuff that is churned out of taiwan or china. But regardless the problem is, as I said before, that the database requires additional hardware (so it would be more expensive to produce the plugin), that we don't even know how the damn thing would be updated, the risk of false positives in the database, and if the game isn't in the database or you get a false positive then you would have to override default plug and play functionality with your own settings, rendering the whole exercise useless since you would need to mess with settings to switch between games, a notion that Matthias was trying to avoid.
Can you state even one flaw with my suggestion?
Matthias_H
April 5th, 2012 at 3:00 am
Guys. This fistfight of yours could amuse a much larger audience if presented in the right venue. Why don't you take it to the forum? ;)
Aaaand here we are.
And here comes his challenger, Muuuuuuuuuuuuuzerrrrrrrrrrrrrr....
(crowd cheering)
I'm just curious as to what skills are needed to take such a project on. Is it just C programming and soldering skills? Am I missing anything?
(Sorry I took so long to reply, but I'm on holiday with limited access - I was about to make a post just yesterday, but my account timed out and I almost lost it. Luckily I grepped through my RAM and the message was almost all still there. Ah, the power of Linux, couldn't have done that easily on Windows :P)
It requires knowledge of how the NES works and ability to code in C to make the software side, and it requires soldering skills to make connector prototypes. Finally, it requires electronics knowledge if you're going to implement a flash chip too, and PCB-making knowledge to make the final design. I don't have many of those ;)
Anyway, back to the database issue - I think it's simpler for the users not to have to know/know how to extract that information from their cartridges. Plus, I believe it was stated recently that the config file is fast running out of room, so even 20 lines for custom games might be too much.
In fact, I would suggest a combination of both solutions - the database-on-a-flash-chip for most games, then your solution for anything not in the DB.
Anyway, Matthias is right - it's all trivial if we can't find someone to actually make the hardware/Retrode firmware. I'm sure I could probably help a little bit but I highly doubt I'd be able to make the firmware side all by myself. As for the hardware side, I barely have a clue. It's the database part that I would be able to do well ;)
Matthias_H: As for solutions for the config file running out of room (if, of course, you need them, I can't remember whether I'm right or not about what you said, I could be imagining things) - how about making it a binary file with a simple open-source editor for it? (Open source so that it can be ported to platforms other than your favourite one ;))
i am not sure how the retrode fw source is written, or if it would even allow it,
but i saw a project once where you select what bits and pieces you need, and click on
download, and it downloads a firmware with only the bits & pieces that you need.
simple to do with #ifdef
//F
Quote from: Muzer on 11/Apr/2012 12:50:10 PMMatthias_H: As for solutions for the config file running out of room (if, of course, you need them, I can't remember whether I'm right or not about what you said, I could be imagining things) - how about making it a binary file with a simple open-source editor for it? (Open source so that it can be ported to platforms other than your favourite one ;))
I always thought it was translated to binary by the firmware. But if it is stored in the raw then no wonder Matthias_H is running out of room. The config need only take up 63 bytes at the most when stored in a binary format.
I second this motion.
Although every minor change to the config would warrant a new editor revision, which needs to be ported to two other operating systems.
A better notion would be if the binary format config was translated to a plaintext config file by the firmware, and then you can translate that backwards into the internal binary format config by the firmware upon saving the text config, assuming there's no OS related issues in doing so. But hey, it could be updated upon hitting reset as a fallback if it can't be updated upon saving the text config.
The config is stored in binary but buffered in text form for the whole trick to work at all.
Might just buffering one line at a time work, assuming no line is dependent on any other?
Quote from: Muzer on 14/Apr/2012 02:40:13 AM
Might just buffering one line at a time work, assuming no line is dependent on any other?
None of the lines appear to be dependent on each other, and the longest line in config is 68 bytes in length (or 70 if you include linebreak chars) if that helps.
Even so, a single line of text could still span multiple sectors on the storage medium. Believe me: there are reasons why it is the way it is :)
Use buffer to generate a comment, setting value (or a single part of a multi value set), setting header, or whitespace one at a time, commit it to the end of your virtual config file (is it loaded into ram when the file is read, or when the USB device is read, or both?), then clear the buffer of any data that isn't required to generate the rest of the config, then repeat for the rest of the config.
Of course I am making assumptions about how this all works.
I'd rather wait with such fundamental changes to the config handling until there's an actual reason for it -- i.e., until someone actually builds a NES plugin.
What about an approach that has been done with the kazoo/unagi?
As I understand it a script describing the attibutes of the mapper/chips is fed to the unagi application which through the kazzo hardware dumps the contents.
http://sourceforge.jp/projects/unagi/wiki/kazzo_intro_en
Just wondering, is it impossible to just dump everything that's found within each cartridge?
I know the PRG and CHR dumps alone wouldn't work on emulators due to the lack of a header, but that could be added later with the appropiate programs, or just implementing all the database stuff to the emulators that want to be compatible with this.
The problem is bank switching. Not all memory segments are "visible" at a time, and without knowing the mapper chip used on the cart you can't access the entire ROM...
Oh, that makes sense... But how could a real NES see all the banks without knowing the mapper in advance, then?
Oh, the NES is completely stupid with that regard. It's the code on the cartridge that takes care of the bank switching by making the NES read/write certain addresses and registers whenever needed. Even if the Retrode were capable of executing console code, this wouldn't give a coherent ROM image without knowing what these registers do.