Some Caveats You Missed in Stirk

Public discussion of the Province: Cyrodiil project.
User avatar
jazzon
Pleb
 
Posts: 22
Joined: 17 Sep 2014, 02:42

Some Caveats You Missed in Stirk

Postby jazzon » 19 Sep 2014, 11:47

First let me say I am not trying to say you folks have done anything wrong. Second, some of this is just opinion, and not fact. Where is fact I will say so, and if you like provide proof.

I just attained Stirk. And aside from some real obvious issues thought it was pretty good. But there were some issues which you either didn't think about, or didn't know about as well.

1 The First issue is folder/file structure related, and is caused by the age of the Morrowind. Some background is needed here so here goes.
    Morrowind was written for either Windows 95 or 98, I don't recall which. Both of these operating systems (at least until one of Win 98's later patches) couldn't tell the difference (internally) between upper and lower case file/directory names. Also they couldn't handle names in non 8.3 format well (that means names with more than 8 characters in the name, and three in the extension) Part of that format was large collection of characters (including spaces) which could never appear in file/directory names. They found clever ways of behind the scenes processing of these things so that users had the illusion that they could use these things, but they really weren't.

    Because of this Bethesda very wisely taught their software to ignore case altogether.

    Now days this is an issue. If a program looks for "c:\Program Files\Whatever\some.fle" it wont see "c:\Program Files\Whatever\Some.fle" (capital S on Some). In the old days it would. And Bethesda expects everything lower case, but somehow finds some upper case things as well. The issue comes up however with mods, especially with more than one person working on them.

    Not all file compression utilities work the same. Today most versions of windows automaticly show the first letter of a directory name as a capital letter. Some file compressors assume that sincs they were told (by the OS) it was capital, then it IS capital on the drive. So when they build the compressed file, the directories all get capitalized. And then someone in Europe works on the mod. Due to the many different languages there, Windows automatically comes with UTF-8 support turned on. Capitals are capitals, and other symbols are handled correctly. This person then packs it up and sends it back to the first person. The first person unpacks it, fires up the game and gets some complaints about missing textures or models.

    The problem was that the first person had a non utf-8 enabled machine. Their OS told the compression program that "c:\Program Files\this directory\" was actually "c:\Program Files\This Directory" (because windows can lie about that stuff, and does frequently). So when person one sent it to person two, all first letters were capitalized in the compressed file, but not on the drive. So when His NON-LYING OS packed it up and sent it back, they also were capitalized in the file, but this time flagged as utf-8. So when person one unpacked the new version he now has "c:\Program Files\some directory" AND "c:\Program Files\Some Directory". Imagine the havoc this would cause in "...\Morrowind\Data Files\*"

    Since Morrowind is played today on many versions of Windows (some of them virtualized), Apple's many OS * versions, Linux, BSD, X-Box, etc. This issue is usually addressed in advance by the better modders. (I myself haven't owned a Microsoft product in over 10 years, it's all Linux)
Long and the short of it: use lower case names only. It doesn't matter what you use in the CS (it ignores case), but make em lower case on the hard drive.

You guys basically beat this problem using a BSA file. But that introduces three other problems.

2 The issues with BSAs
a) Any file with the same name and path on the hard drive automatically overrides your file, even if it's from another mod. (BSA files do not record the date a file was created, so the assumption is that any file on the drive is newer)
b) BSA files are completely stored in memory at run time. This makes cells load quicker, but limits the number of mods the user can run. (memory fills up and out of memory errors and CTDs happen more frequently)
c) BSA files require an entry in morrowind.ini, and not every user of your mod will know how to do this, or do it right, or be comfortable doing this.
d) Most serious players unpack them anyway so using them is a waste of your time.
e) When Morrowind throws errors about bad/corrupt/missing files (and they're in a BSA) the messages are useless to the average person. They go to look for the file, and assume it is missing. You thus waste a lot of forum time addressing issues of the uninformed, or solving the issues of some one else's mod, and not yours in the first place (see the issue below).

3) Directory names
Simply put there are at least two other modders out there that I know of that use PC as their directory. I would suggest "pcyrodil" or "p_cyrodil". (When I installed Stirk the file compressor bitched about overwriting certain files in PC directories on my drive. Since I plan on a fresh install soon any way, I didn't pay attention to which mod so I cant tell you which, but I will try to find out.

4) Archive structures.
You folks didn't do this, but if you choose to take my suggestion and go to loose files you might. Do not include the "Data Files" directory in your mod structure. Most people who manually install a mod simply place the compressed file in their "Data Files" directory, and do a "Unzip here" or "extract here" command.

5) Documentation
Use a "Docs" folder (belongs inside "Data Files"). People like this. And name your documents like "[name_of_esp_or_esm_file_here]-readme.txt" (without the [ and ] obviously. This is because most people who use lots of mods use WryeMash to manage them. If your txt files are not in a Docs folder it will put them there anyway (this one is deliberately capitalized). And if you have a generic "readme.txt" it will be overwritten by the next guys. And most people cant remember mod names (if they have a lot of them) so they look for the esp/esm name as part of the docs.
Also use underscores instead of spaces (there are still a few obscure OSs in use that don't like the spaces in file names or directory names. My personal choice is underscores within a name part, and dashes to sepperate parts. For example "my_mod-v3_4_76-readme.txt" Yeah it's odd, but so am I. (Also it's what most pro programmers do, so you look more (for lack of a better word) credible.)


Again, I'm not trying to tell you folks your business. Just offering up some of my experiences. Grumpy and I took several hours one day trying to figure out the whole Upper Case / lower case thing many years ago after I did an Win 98 upgrade and my mods mysteriously began to break.

Simply trying to help you folks avoid issues in advance.
I have been accused of writting walls of text ... this is completely and utterly false!
I have constructed many Great Walls however ....

I do post long things, so I use spoiler tags as condensing blocks. Open at your own risk!

User avatar
Scamp
Head of Reviewing
 
Posts: 1192
Joined: 17 Feb 2011, 19:38
Location: The Gold Coast

Re: Some Caveats You Missed in Stirk

Postby Scamp » 19 Sep 2014, 14:19

Hey, thanks for your input.

Regarding BSAs:

a) Do you really think someone is going to have "pc_ww_ground_dirt_01_leafy.dds" in their mod? Highly unlikely.
b) Just so that we are on the same page, where did you get this information from?
c) We have extensive guides on how to handle this. Anyone who can't handle this after reading these guides clearly should go back to school instead of playing video games.
d) I know nobody who unpacks BSA files just for the sake of doing it. Why would you do that? Most modders nowadays use Tamriel Rebuilt, which uses a BSA archive, so they are familiar with how BSAs work and I've never heard of anybody unpacking the BSA as it makes no difference ingame.
What do you mean by "serious players" anyway? I consider myself a serious player. The only reason why you would unpack a BSA is because you want to make changes to meshes and/or textures. Very few people actually do that.
e) There shouldn't be any corrupt files in our BSA after release. We test our releases extensively.

Honestly, having all the Meshes and Texture folders around is a giant mess. BSAs seem fine to me, never heard about memory running out or whatnot. Have you experienced anything of that kind? Also, memory is soooooo cheap nowadays. I have 8GB and I barely even had to pay anything for it.

Even if other mods use the PC directory, what's the issue with that? Your operating system likely bitched about the folder having the same name, but it's (as I've said before) unlikely that other mods will use the same file names we are using - our file names are quite long - and thus, simply combining the two folders shouldn't prove too difficult.

Anyway, thanks again for the input, not trying to be rude at all, just my two cents on some of your points.
Image

User avatar
jazzon
Pleb
 
Posts: 22
Joined: 17 Sep 2014, 02:42

Re: Some Caveats You Missed in Stirk

Postby jazzon » 19 Sep 2014, 15:27

@Scamp
a) Just had it happen when I unpacked your BSA on my system (some file 2 I think in PC folder). I grant this is not a likely thing to see, but it did happen as I unpacked yours.

b) The information about windows 95/98 vs new operating systems can be found in the Microsoft web site under the section on long file names (in older versions, not sure if that's there any more). They used to convert spaces to underscores behind the scenes, and truncate directory and folder names like this: C:\Program Files [this visible to user by typing dir on a command line or viewing in explorer.exe] internally this was C:\Progra~1 [only seen by user by using dir on the command line with the /all switch appended]) Most of this is common knowledge among programmers who were around and coding during the period. If you had a c:\reallylongnameone and a c:\reallylongnametwo, then behind the scenes they were c:\really~1 and c:\really~2 (first six characters, a tilde and then a count). The count would handle ... imm not really sure how far they would count up top be honest...I never checked.

c) Morrowind guides were (mostly) written during morrowinds heyday. Ghan's scripting bible made it version 8 way back when, and wasn't looked at again (although there is an unsanctioned version avail on the www) Most of these problems couldn't have existed back then because there was only windows and mac in peoples homes. And those two could not even talk to one another.

d) Most people that I know use BSAUpacker or something similar to unpack BSA files for their games which use them (primarily Bethsoft since it is their format). And they do this for three reasons mainly 1) Faster game load 2) Easier debugging 3) easier to manipulate should they choose to make personal modifications of the files. The trade off for this is slightly longer cell load times. In my case (with over 200 mods loading) I see the loading bar for less time than it would take to speak the word "one" aloud.

e) That part was poorly phrased. Another modder has a PC folder. Their mod has a bad file. The error messages in game don't say which mod had the error, usually you can just see the tail directory and the file name. Most users don't read "warnings.txt" They think "PC=ProvinceCyrodil" and spam you people with messages you shouldn't have to deal with.

f) As for the file handling issues of file compressors, I ran the tests myself less than a month ago when I started to see PC and pc as directories in teh same folder, and all I had done was update the same mods I already had. Since I only use one file compressor, something had to vchange on the other end, but I tested any way. Tested RAR, an old copy of WinAce, WinZip, 7-Zip, Ultra-Zip, and all of the various linux compressors. I don't remember which did what by default. But I got three different types of handling. True respect for characters, First letter capitalized without needing to be, and case ignoring. Just to ensure my tests were valid, I tried with multiple compressed files, some compressed on windows then unpacked here (had the most trouble with those since UTF-8 is not ACTIVATED by default on some windows installs even though support is there), some from a Mac, and some from Linux. I only own my Linux based machines, so I could not test differences in unpacking on differnt OSs. Since Linux and all Macs share much core code, their results should be similar. Windows I cant speak for at all. The one Mod I tested and got 2 differing results from the same file that I can remember off hand is Better Bodies 2.2.

As for memory, Morrowind is written for 32 bit machines. Without a special patch anything above 2 GB is unknown to morrowind. I personally have mine patched to 4 GB and Im running 16 GB so I'm not the one to ask about this now days. Years ago though the second biggest CTD error was memory. Of course back then having 512MB of ram was a true monster machine.

I meant no offence to anyone by posting this. What you guys and TR (I havent looked at this SHOTA ? SHONTA yet) are doing is great! I want you to succeed! I just wanted throw the problems I am aware of that can come up at you. Forwarned is for armed after all.

(I just wish Microsoft shipped their European version of windows in the US, that alone would solve many programming problems)

[EDITED: Tried to space it out some...too big a wall to read easily]
Last edited by jazzon on 19 Sep 2014, 16:11, edited 1 time in total.
I have been accused of writting walls of text ... this is completely and utterly false!
I have constructed many Great Walls however ....

I do post long things, so I use spoiler tags as condensing blocks. Open at your own risk!

User avatar
jazzon
Pleb
 
Posts: 22
Joined: 17 Sep 2014, 02:42

Re: Some Caveats You Missed in Stirk

Postby jazzon » 19 Sep 2014, 16:09

OK, I went looking for info to get you the data on MSs web site. Since they no longer support 95/98/NT/ME/XP etc. the docs are gone but here is some other data I found on name mangling (the actuall name for this knid of crap)
Form wikipedia at this page
Probably the best-known example of name mangling occurs on VFAT file systems on versions of Windows from Windows 95 onwards. The VFAT specification allows Long File Names (LFNs). For backwards-compatibility with MS-DOS and older Windows software, which recognizes filenames of a maximum of 11 characters in length with 8.3 format (i.e.: an eight-letter filename, a dot and a three-letter extension), files with LFNs get stored on disk in 8.3 format, with the long file name stored elsewhere on the disk. Normally, when using compatible Windows programs which use standard Windows methods of reading the disk, the I/O subsystem returns the long filename to the program — however, if an old DOS application or an old Windows application tries to address the file, it will use the older, 8.3-only APIs, or work at a lower level and perform its own disk access, which results in the return of an 8.3 filename. In this case, the filenames become mangled by taking the first six non-space characters in the filename and adding a tilde (~) and then a number to ensure the uniqueness of the 8.3 filename on the disk. Thus this mangling scheme can turn (for example) "Program Files" into "PROGRA~1". This tradition continues today when people use DOSBox to play classic DOS games or use Windows 3.1 in conjunction to play Win16 games on 64-bit Windows.


Wow! Wikipedia is the only place left that mentions it directly (quick search only not in depth at all.)

Alright fond another at this site. Its a mailing list server the topic being discussed is about how a version server (for authoring software among a group of people) has to deal with these issues even today. It's not real specific but if you look up Mercurial (the software they are discussing) You will see it is one of the big boys in the industry (comparatively), and this is an issue even for them.

Turns out (and I should have realized this) that the whole cap/no caps/and truncating thing is also do to the file-name mapping table inherent in any file system specification. For non programmers that refers to how the drive is formatted (FAT/VFAT/FAT32/NTFS/many many more) AND the default character encoding of the Operating System. As such, modern file systems aren't a problem. So now days teh problem is all in UTF-8 support being ACTIVE not just installed. Anyone whose system has UTF-8 (or higher) as their default (Macs, Linux, Windows Pro versions) will se the file names as they truly are. Others (Basically non-pro versions of windows) may or may not run into this problem, depending on the settings of the machine that compressed the file.

Anyway, hope this helps clarify what I said before. I meant no offence. I am the "old man in the room" I am sure. Just trying to help you "youngsters" avoid teh issues I've encountered and seen. Most people have to run Morrowind in Dosbox, so it needs to be treated as an OLD windows software wants to be treated. Which really sucks for those of us that still love it!
I have been accused of writting walls of text ... this is completely and utterly false!
I have constructed many Great Walls however ....

I do post long things, so I use spoiler tags as condensing blocks. Open at your own risk!

User avatar
jazzon
Pleb
 
Posts: 22
Joined: 17 Sep 2014, 02:42

Re: Some Caveats You Missed in Stirk

Postby jazzon » 19 Sep 2014, 17:18

HA HA! FOUND IT AT LAST
Straight from Microsoft.
A knowledge base article about why 32 bit windows applications (Morrowind) have trouble with NTFS

Read the whole article.

The issue is caused byt 32 or 64 bit system on the compressing side. UTF-8 Unicode support is also involved but not discussed here.

But it is Here!

In a nut shell, UTF-8 or better is NOT enabled by default on ANY windows platform. It MAY be enabled by application settings, but not for all applications globally. (found by reading several articles there)

(I hate to admit this sionce I dislike Microsoft in general, But I have to applaud their early adoption of UTF-16)
grumble....mumble....hssssss......good job guys......hssssss....mumble...grumble
I have been accused of writting walls of text ... this is completely and utterly false!
I have constructed many Great Walls however ....

I do post long things, so I use spoiler tags as condensing blocks. Open at your own risk!

User avatar
Mwgek
P:C Reviewer
 
Posts: 111
Joined: 13 May 2012, 13:13
Location: Netherlands

Re: Some Caveats You Missed in Stirk

Postby Mwgek » 19 Sep 2014, 20:22

BSA is fine. No one in the morrowind community wants the shitload of models and textures in there data folders. Only people who want to mod the files, like myself, have the unpacked bsa's from the province mods on a hard drive... BSA keeps your data files clean.

User avatar
Scamp
Head of Reviewing
 
Posts: 1192
Joined: 17 Feb 2011, 19:38
Location: The Gold Coast

Re: Some Caveats You Missed in Stirk

Postby Scamp » 19 Sep 2014, 22:17

Morrowind guides were (mostly) written during morrowinds heyday.


I am not talking about this kind of thing. I am talking about our own instructions that come with the release (if they don't, that's bad. I remember we have it somewhere, at least TR does).

Just trying to help you "youngsters" avoid teh issues I've encountered and seen.


Eh... not sure why you judge us and our knowledge about computers without really knowing anything. Sorry if that sounds a tad harsh.
Image

User avatar
jazzon
Pleb
 
Posts: 22
Joined: 17 Sep 2014, 02:42

Re: Some Caveats You Missed in Stirk

Postby jazzon » 20 Sep 2014, 02:33

LOL!
OMG I cant seem to get this to come out right!

That wasn't meant as a judgement, but as a humorous aside. I am a whole lot closer to 50 than I'd like to admit, the only "judgement" I'm making here is the assumption that I'm "the old man in the room" as it were.

You guys obviously know what your doing if you've gotten this far. I was simply trying to point out potential pitfalls that are easier to fix in advance rather than solve later.
I have been accused of writting walls of text ... this is completely and utterly false!
I have constructed many Great Walls however ....

I do post long things, so I use spoiler tags as condensing blocks. Open at your own risk!

User avatar
Scamp
Head of Reviewing
 
Posts: 1192
Joined: 17 Feb 2011, 19:38
Location: The Gold Coast

Re: Some Caveats You Missed in Stirk

Postby Scamp » 20 Sep 2014, 15:35

Haha ok, I do overreact at times especially when it comes to misunderstandings.

Like I said, your input is appreciated and we can put some of it to good use; I really don't see a problem with BSAs, however.

If you could find out what mods are causing these conflicts btw, that'd be valuable information as well.
Image

Worsas
Asset Manager
 
Posts: 847
Joined: 17 Mar 2011, 20:55

Re: Some Caveats You Missed in Stirk

Postby Worsas » 21 Sep 2014, 22:17

You are probably more adept at pretty much everything regarding software but I found one wrong statement, though:
a) Any file with the same name and path on the hard drive automatically overrides your file, even if it's from another mod. (BSA files do not record the date a file was created, so the assumption is that any file on the drive is newer)

If the BSA-File has a newer date than the file in the directory, the file inside the BSA will be used. I have had issues using MGE where MGE would render the model according to the model found in the directory while the model from inside the BSA was used in MW-viewrange (the MGE-developers seemingly assumed that the files outside of the BSA would always get preference, but that is not the case).

It is sometimes an issue, when I install a new version of PC_Data with a new BSA-File but I have an alternative version for something in PC_Data.bsa in the directories. I only get to see the version from the directory to show up again when I set the change date to newer than the date of the BSA.


Return to P:C Public Discussion



Who is online

Users browsing this forum: No registered users and 1 guest

cron