Hi,
I have, along with Wannado have been doing some research into why the SRM files for Megadrive games are "disappearing"
It's possibly due to the emulators saving the games in different sectors of the SRM files. So when you load your save file in a different emulator such as Kega, the emulator will search for where it thinks the save data should be but instead is greeted by 00x00 instead of whatever the save information should be. We just need to find the correct sector & stop the retrode from replacing the save file with a new blank file.
Here's what Wannado says: I'm guessing the following: In the Retrode file, the data repeats after 16 kiB. This is also known as a mirror, because both copies of the data come from the same location in memory.
The emu loads the whole 32 kiB, but the game only modifies the first 12 kiB. The emu still saves the whole 32 kiB, because it cannot be sure that truncating the file would be safe. The result is that the save on the cartridge is overwritten twice: First with the new data (first 12 kiB), then with the "second copy" of the old data starting at the 16 kiB offset. The latter "restores" the old save. (The 4 kiB pieces at the 12 kiB and 28 kiB offsets are probably lost as they are beyond the end of cartridge memory.)
I have tested it with Kega, and found that the SRM files differ depending on the type of emulator you use. I have a feeling that the Retrode itself wipes out your original save upon connection & replaces it with a blank save instead which doesn't help matters.
However, if you create a save with a specific emulator, say like KEGA save that, copy it over. let the Retrode wipe out your save, copy back the kega save, then reload the emu. The previous non existing save will reappear. All we have to do as Wannado puts it is detect the correct save size for each game.
I think that with Wannado & help from the community we are getting closer to solving this issue. :D
Upon further inspection, my hunch that the retrode does indeed create new saves was correct.
If you look at the dates the Retrode reports, they change from (fresh save) 4/7/2016 > upon disconnection /re connection 20/11/1990 along with an altered time of 23:00
Plausible explanation, except that the Retrode doesn't "create new saves" - if anything, the OS does. For various reasons, the Retrode's entire file system is made up on the fly, and a change in the file date doesn't have anything to do with the question whether it's an old or new file.
If thats the case, then surely there must be a way to stop the OS from creating a new file. As ive noticed with a hex editor, upon re connection the top of the file reports all FF's which from what it looks like is that the OS has blanked out some important information.
Where as a file which i replaced & i know that works, it has FF 00 FF 00 FF 50 FF 48 FF 41 FF 4E FF 54 FF 41 FF 53 FF 59 FF 20 FF 53 FF 54 FF 41 FF 52 FF 33 FF 65 FF D6 FF 49 FF 00 FF 52 FF 68 FF 79 FF 73 FF FC FF 00 FF 29 FF 00 FF 00 FF 00 FF 07 FF 00 FF 00 FF 04 FF 8F FF 01 FF 10 FF 01 FF 40 FF 00 FF 00 FF 01 FF 80 FF 01 FF 00 FF 00 FF 80 FF 02 FF 00 FF 02 FF 80 FF 00 FF 04 FF 00 FF 02 FF 00 FF 00 FF 00 FF 06 FF 00 FF 08 FF 29 FF 48 FF (which comes from Phantasy Star IV) when saved with Kega Fusion.
When this is loaded, up comes the save file. I don't have a way to edit the SRM, to test if this is of any use. Also if you look that in a hex editor, it comes up with jumbled up lettering.
When i spoke to Wannado, he did mention that i had a point about the size of the save file not being the right size. & that could well pose an issue as to why the saves are not working. Either way, i think we are getting closer to the retrode being cross platform emulator / megadrive. :D
My statement that gliitch quoted in the OP was a response to a PM not quoted here. The numbers given may be wrong, but the general idea may still be right.
In the meantime, gliitch has sent me a few sample save files (thanks!), and I have taken a look at them. Those save files have been created by different means, but belong to the same game (seemingly Phantasy Star 3 - Generations Of Doom).
1.
From "everdrive":
I guess this file has been created playing the game using an Everdrive plugged into a MegaDrive, and retrieved by reading the Everdrive's storage media with a regular card reader. Hence there was no interference from the Retrode.
@gliitch: Is that correct?
The file size is 12 kiB, but the first of every two bytes has the value FF. Removing those FF bytes yields 6 kiB of data resembling the other save files provided:
* 2 kiB savegame.
* 2 kiB unused memory (all FF).
* 2 kiB savegame identical to the first one - @gliitch: Did you store the same game in two slots (I'd have), or might this be some kind of mirror?
Note: I guess that this is followed by another 2 kiB of unused memory, for a total of 8 kiB of memory, but the Everdrive truncates the file to the size that is actually used.
2.
From "megadrive":
I guess this file has been created using the original game cartridge in a MegaDrive, and retrieved using the Retrode.
@gliitch: Is that correct?
The file size is 16 kiB minus 1 byte (why?). The data structure is:
* 2 kiB savegame.
* 2 kiB unused memory "A" (filled with alternating patterns of FF FF ... 00 00 ... disturbed by very few different, "random" values).
* 2 kiB savegame (identical).
* 2 kiB unused memory "B" (like above, but the "random" values and their locations are different).
* 2 kiB savegame (identical).
* 2 kiB unused memory identical to "A".
* 2 kiB savegame (identical).
* 2 kiB - 1 byte unused memory identical to "B", except that the last byte is missing.
3.
From "retrode":
I guess this file has been created using an emulator (Kega Fusion), the Retrode, and the original game cartridge.
@gliitch: Is that correct?
The file size is 16 kiB minus 1 byte. The data structure is:
* 8 kiB filled with alternating patterns of FF FF ... 00 00 ... disturbed by very few different values.
* 8 kiB - 1 byte identical to the first 8 kiB, except that the last byte is missing.
So, compared to the "megadrive" file, the "retrode" file has a similar structure, but the save slots seem to be empty.
Based on the above, my current, still flawed theory is:
The game has 8 kiB of memory used for two save slots. Or maybe four save slots, and gliitch likes to use only the first and third of them, for a game and a backup copy. :) Or the game itself duplicates the save slot(s) because there is such an abundance of memory, and if one copy is corrupted, the other may still be valid.
For some reason, the Retrode presents a 16 kiB save file. The first 8 kiB (16 sectors) of the file are mapped to the save memory as intended. But when the Retrode tries to access the nonexistent second 8 kiB of memory, the cartridge ignores the highest 1-bit of the address and again returns (when reading) or modifies (when writing) the data in the first 8 kiB. So the second 8 kiB of the file are mapped to the same memory as the first ("mirror").
See also my statement in the OP, with 32 kiB / 12 kiB replaced by 16 kiB / 6 or 8 kiB.
The flaw in this theory is: Above, gliitch said that the Retrode "wipes out" the save, replacing it with a "blank" one (like 3. above). But in my theory, the save should only be reverted to its previous state, not vanish completely.
This would be consistent if the "previous state" was "blank", but I have no idea how gliitch could have created such a state after playing the game on the MegaDrive. Note the imperfect FF FF ... 00 00 ... patterns: if the game had intentionally cleared the slots, it would likely have filled them with all 00 (or all FF) instead.
I don't think that the battery is dead, because gliitch would likely have noticed that, and creating the "megadrive" file (2. above) would have failed then.
I don't know how the transition from "console power" to "battery power" works (or if there is a transition at all), but maybe the Retrode is controlling it wrongly, causing a short blackout in the save RAM.
Any ideas / comments?
@Wannado, to save space on this thread, all of the question that you have raised, you are indeed correct. I am not too well versed in data structures. I will help where i can. It certainly seems as though you have a far better understanding of these values and being able to create bug fixes (take the last firmware update for example) then i do, it would be awesome to learn a few new things :)
Glad there's still work ongoing on this issue. It's one of the biggest uncorrected flaws of the Retrode IMO. Basically I believe it's impossible to overwrite Genesis SRM's that have been altered through an emulator. The cart will always retain it's previous SRM unless moving back an SRM that came off a cart and hasn't been altered in an emu. The other problem is many Genesis emus use different save files. I've seen 16kb, 32kb and I believe even 64kb's with some emus. Sometimes they work between each other and convert the file sizes to the newly used emu. In other words it's a bit of a mess. :/
I haven't looked it up, does Retrode support Phantasy Star IV correctly in the first place?
As I have read, PS4 uses a mapper where the first 2MB of ROM is always mapped in, and then the last 2MB of Genesis cartridge space can map in either of the last 1MB ROM or the SRAM. (and I heard that at one point the game does a copy-protection check by mapping in ROM and attempting to write to it)
I believe it works but somehow it deleted my save file in slot 1 and 3. Fortunately all three were late game saves past a ton of grinding but since I hadn't played the game in almost 15 years I can't remember which was the best slot. Anyway all I had left was slot 2 after copying the save. I didn't try saving in 1 and 3 to try again. If anybody has please share the results.
I found a possible explanation for the issue: The cartridge's address bus does not have a line for the least significant bit (LSB) of the byte address. Instead, its data bus is 16 bits wide, so that a two-byte "word" can be read at once.
Hence the Retrode has to divide the byte addresses by two to form word addresses, and use the byte address LSB to select which half of the data bus to read from/write to. Also, some cartridges connect only one half of the data bus to SRAM, and emulators disagree about how to handle this. That's why there is the segaSram16bit setting in RETRODE.CFG.
And the bug might be here: The ROM and SRAM sizes are computed from the respective start and end addresses given in the ROM header. I think that all four addresses are byte addresses. But the Retrode firmware seems to treat the SRAM limits as word addresses, so that the computed SRAM size is two times the real SRAM size.
The next release, v0.20 beta, will contain an attempt to detect what the unit of the SRAM limits in the ROM header really is, and to compute the size accordingly. (The Retrode expects the SRAM limits to fall within in the byte address range 0x200000 to 0x20FFFF, which equals the word address range 0x100000 to 0x107FFF.)
Update 2016-10-29: Logfiles gliitch sent me indicate that the SRAM limits are stored in the header as byte addresses, just like the ROM limits.
Wow I had ended up thinking we'd never actually get this issue resolved someday. Great work Wannado, as always you're the key to keeping this awesome project alive!
If only I was keeping it alive faster. ;)
I don't have any MegaDrive cartridges to test if this really resolves the whole issue. But at least it is consistent with my theory about the mirroring.
Just in case, the release will also include a firmware variant that produces a logfile with information that might help in the analysis of this issue.
Great. Can't wait to try it out. :D
I sent it to Matthias two weeks ago, but maybe it was lost or he just didn't have time to upload it, so to speed things up, I'm posting it here (see attachments).
See the enclosed readme.txt for details.
Note that I can only perform few tests on each release. In this case, I didn't even have the means to verify that the bug fixes had their intended effects. I only tested that a few other things still work as before.
I couldn't test some of the log messages I added, so the logfile variant might freeze or otherwise misbehave.
Update 2016-10-29: Meanwhile, gliitch and Pickle have performed some tests, see reply #14.
Update 2016-11-14: WARNING: The computed SRAM size is still wrong, see reply #19 (http://forum.retrode.org/index.php/topic,337.msg2361.html#msg2361).
Thank you for the new version!
gliitch has performed quite a number of tests of v0.20 beta with Phantasy Star III, Phantasy Star IV, and Shining Force.
Thanks a lot, gliitch!He reports that the savegame file (*.srm) mostly works (but see restriction below):
- The SRAM size
seems to be detected correctly now (see also Reply #8 (http://forum.retrode.org/index.php/topic,337.msg2316.html#msg2316) above). Update: It's still wrong in many cases, see reply #19.
- The file can be copied to and from the cartridge properly (see requirements below).
- Direct access from emulators (including on Android) also works (see below).
Requirements:
- The game/console region may be relevant for savegame compatibility. This is an issue of the games/emulators, not the Retrode.
- It only works with firmware v0.20 beta or later.
- You must set "[segaSram16bit]" in RETRODE.CFG to 0 or 1 depending on the format used by the emulator / of the file to be copied.
I guess a simple rule of thumb is: If the file sizes are different by a factor of 2, toggle the setting. I recommend that you copy the file from the cartridge as a backup before and after toggling the setting. The smaller backup might be useless - keep both. - By default, the Retrode's savegame file is write protected. To disable the write protection and be able to modify the on-cartridge data, you must set "[sramReadonly] 0" in RETRODE.CFG.
Restriction:
- There still seems to be an issue with the third save slot of Phantasy Star IV and Shining Force. It might be wiped or inaccessible.
Thanks also go to Pickle!Pickle has tested the combination of "[forceSize] 8" and "[forceSystem] GG" with v0.20 beta and reports that it worked. (Said combination indicates an 8 megabit Game Gear cartridge.)
I have just tried testing saving to an EU cart from an US rom, with Phantasy Star IV. It works, & reads back as if you were playing from a cartridge from your own region. I think its because it's copied straight to the cart itself, without any checks, so provided you have a matching SRM name ie: PhantasyStarTheEnd7FC0 which then matches the bin file on the cart itself, i dont see any reason as to why it wouldn't work.
I also got a downloaded save to work, turns out i don't think i left it long enough in the retrode to copy properly :-[ But its awesome that we finally have saving SRM working :D also, you can use games with game genie codes & it will still save your progress, you'll just have to reinput the codes in the emulator.
Oh my gosh. I can't wait to try this out. This sounds like the holy grail of updates for Genesis/Megadrive fans such as myself!
What file sizes were you working with gliitch? Real Genesis srams appear to be 32 but emulators I use give me 64 kb srams. So does the new update allow the Retrode to correctly rewrite the file size into a 32kb sram?
For what it's worth I've encountered that Phantasy Star IV (US copy) problem back when I first bought the Retrode. I thought it was my battery that was dying, I had backed up my games prior to replacing all the batteries. What happened was save 1 and 3 were wiped. Fortunately all three were close to each other so not much was lost with just the second save. I've never had any other oddity like that with the retrode on the SNES or any other Genesis game.
The new update indeed allows for the carts to be read properly, there are still some issues (3rd save slot gets wiped) 1 & 2 are fine, Wannado is on the right track, i can't really speak for genesis owners such as yourself, but for me Phantasy Star 3 / 4 & Shining Force save as they should. The save issue isn't just specific to Phantasy Star 4 but also Shining Force also.
I don't see why Genesis carts would be any differrent. I thought it maybe an issue with the size of the game but Wannado, who is far more knowlegeable about these things would know. He will probably ask you to send him some Genesis save files.
SNES were working from the start i think, at least thats what most people said, it was always just the megadrive that posed a problem. Let me know how it goes for you, as I've only got those 3 games that have SRAM saving, it would be interesting to know if Sonic 3 works, that is if you have it? :)
Quote from: Nori on 30/Oct/2016 08:18:05 PM
What file sizes were you working with gliitch? Real Genesis srams appear to be 32 but emulators I use give me 64 kb srams. So does the new update allow the Retrode to correctly rewrite the file size into a 32kb sram?
The cartridge has a 16 bit data bus, but some (many?) games are using only half of it for SRAM.
Most(?) emulators don't seem to care and just pretend that the whole data bus is used. So for each real SRAM byte, they store an unused byte (hex value FF) next to it. This makes the emulator's save file twice as large as the cartridge's SRAM.
In such a case, you must set "
[segaSram16bit] 1" in RETRODE.CFG. For more details, see reply #14 (http://forum.retrode.org/index.php/topic,337.msg2355.html#msg2355).
I just noticed that v0.20 beta still has a bug in the size computation for MegaDrive/Genesis savegame files (*.srm).
The computed size is one byte too small in many cases, so that the last byte of SRAM is not read or written. If the missing byte is relevant to the game, effects may range from subtle changes to wiping a save slot or the whole save data.
For example, this may be the reason why the third save slot of Phantasy Star IV and Shining Force was wiped in gliitch's tests (see replies #14 and #17 in this thread).
Great find. Unlike Glitch, in my tests of transferring an existing srm obtained through an emulator to the cart still doesn't work. Although with v0.20 I now get a message notifying me that the cartridge doesn't have the room for the srm (which is two times larger than regular srm's sitting at 64kb instead of 32kb).
Once again sorry for the delay. I wasn't able to do much in the past few weeks.
Anyway, the emulator that Nori is using seems to always store the full contents of the address range from 0x200000 to 0x20FFFF1, regardless of what part of that range is really populated with SRAM (according to the ROM header). So srm files written by that emulator are always 64 kiB in size.
I'm adding support for this save format to the firmware (will be v0.21 beta). It will be selected by setting "[segaSram16bit] 2".
1 Note that I couldn't find out if there are any (commercial) games that put SRAM outside that address range, or if that would be possible at all. The ROM header would allow for it, and the address range assigned to the cartridge as a whole (including ROM and RAM) seems to be much larger.
Wannado you are amazing. Can't wait to try that out in the next version. :D
Here it is, f i n a l l y.
I only tested it on the Retrode 2 with a few games (GB, SNES) and an N64 Mempak.
As always, see the enclosed readme.txt for details.
Quote from: Wannado on 26/Jan/2017 11:16:33 PM
Here it is, f i n a l l y.
I only tested it on the Retrode 2 with a few games (GB, SNES) and an N64 Mempak.
As always, see the enclosed readme.txt for details.
Fantastic! Can't wait to try it out!
Unfortunately when I try to download the file for the Retrode 2 I get this message:
"File could not be saved, because the source file could not be read."
If I force download it I have it visible on the desktop but it tells me the file is either damaged or corrupted. :(
EDIT: Nevermind, a few minutes later it worked! I'm going to try this out at my first break tomorrow (right about to pass out for the night). :)
It WORKS!
Using SegaSram16bits set to "2" everything I've sent to it worked. You can even change it back to "1", restart the Retrode, and get the save converted to a regular sized file.
It's absolutely amazing. Now the Retrode does both the SNES and Megadrive/Genesis save transferring absolutely flawlessly. It's perfection.
Thank you so much Wannado for your hard work and perseverance at solving what appeared to be, or at least that we thought, was a nearly unsolvable issue. You're the greatest!
So with this issue now solved, what's your next priority with regards to the Retrode? :)