OpenXcom Forum

Modding => Tools => Topic started by: Hobbes on June 24, 2014, 08:19:36 pm

Title: PCK View issues
Post by: Hobbes on June 24, 2014, 08:19:36 pm
I'm getting a lot of error messages using PCKView. At first I thought it had something to do with the graphics using the wrong palette but now I'm more and more convinced that there may be something wrong. Essentially everytime I add additional BMPs or replace existing ones I get the error message attached below. And it actually seems cumulative. Today I rebuilt a PCK file from one of the original files and at first I was getting no errors but now after 150 images replaced I can't seem to do anything. Sometimes the same image will be properly saved and I can load the PCK file afterwards, other times I'll use the image to replace another one and after saving I can't reload it.

If anyone could look into PCKView and try to find out what's happening I would appreciate it. I'm wasting a lot of time with these trial and error experiments since I'm pretty sure I eliminated the possibilities that my .bmps are causing the problem.
Title: Re: PCK View issues
Post by: Falko on June 24, 2014, 09:22:42 pm
sotty i cant help with the issue itself.
but why do you build PCK files in openxcom? the programm can combine png/gif just fine in "virtual?" old/new PCK data
Title: Re: PCK View issues
Post by: Hobbes on June 24, 2014, 09:43:12 pm
sotty i cant help with the issue itself.
but why do you build PCK files in openxcom? the programm can combine png/gif just fine in "virtual?" old/new PCK data

I need to build the PCK files in order to use them together with the MCDs for Map View.

I'm trying to even use MCDEdit to load the PNGs but the images get messed up after I save. Either the PCK files have a maximum number of images they can hold (but I had this behavior yesterday and I managed to avoid it today) or I'm clueless about what's happening.

EDIT: After several retries I'm more inclined to think I've reached some sort of internal limit since it's the best explanation for all the odd behavior I got. My current PCK file has 142 images, while the biggest PCK files on UFO/TFTD have 133 images so it's likely that's the explanation.
Title: Re: PCK View issues
Post by: davide on June 25, 2014, 01:19:09 am
I am available to debug PCKView

I need your files and istructions to reproduce the issues

Title: Re: PCK View issues
Post by: Hobbes on June 25, 2014, 01:37:20 am
I am available to debug PCKView

I need your files and istructions to reproduce the issues

Thank you for the assistance :)

I've attached a .zip file containing the PCK/TAB/MCD files and 4 BMP files I was trying to add.

Essentially open the AREA51Z.PCK file and add (not replace but add a new image) one or more of the .BMP files and save it under PCK format under a different name. Afterwards open the new PCK and you should get the error message I previously attached.

If you have MCDEdit try opening the new AREA51Z MCD file and check how the images start getting garbled.
Title: Re: PCK View issues
Post by: davide on June 25, 2014, 02:12:25 am
I reproduced it
Exception was fired in this methods, there is a possibility that  the problem is in the TabFile
The exception was fired at the end, I add a comment

Code: [Select]

public class PckFile:XCImageCollection
{
private int bpp;
public static readonly string TAB_EXT = ".tab";

public PckFile(Stream pckFile, Stream tabFile,int bpp,Palette pal,int imgHeight,int imgWidth)
{
if(tabFile!=null)
tabFile.Position=0;

pckFile.Position=0;

byte[] info = new byte[pckFile.Length];
pckFile.Read(info,0,info.Length);
pckFile.Close();

this.bpp=bpp;

Pal=pal;

uint[] offsets;

if(tabFile!=null)
{
offsets= new uint[(tabFile.Length/bpp)+1];
BinaryReader br = new BinaryReader(tabFile);

if(bpp==2)
for(int i=0;i<tabFile.Length/bpp;i++)
offsets[i] = br.ReadUInt16();
else
for(int i=0;i<tabFile.Length/bpp;i++)
offsets[i] = br.ReadUInt32();                                      https://STEP HERE
br.Close();
}
else
{
offsets = new uint[2];
offsets[0]=0;
}

offsets[offsets.Length-1] = (uint)info.Length;

for(int i=0;i<offsets.Length-1;i++)
{
byte[] imgDat = new byte[offsets[i+1]-offsets[i]];

for(int j=0;j<imgDat.Length;j++)
imgDat[j] = info[offsets[i]+j];                  https://EXCEPTION: bpp = 4, i = 0,  j = 0, offset[0] = 34734080

Add(new PckImage(i,imgDat,pal,this,imgHeight,imgWidth));
}

pckFile.Close();
if(tabFile!=null)
tabFile.Close();
}

now i am sorry but i must go to sleep
tomorrow I will check it better, before I must study MCD/PCK file format to understand the problem
Title: Re: PCK View issues
Post by: davide on June 25, 2014, 02:30:36 am
If you have MCDEdit try opening the new AREA51Z MCD file and check how the images start getting garbled.

I have exe of MCDEdit 1.2 by Volutarr, it run but  i am not able to use it:
 I dont found menu to open a PCK :-[

Title: Re: PCK View issues
Post by: Hobbes on June 25, 2014, 02:38:18 am
I have exe of MCDEdit 1.2 by Volutarr, it run but  i am not able to use it:
 I dont found menu to open a PCK :-[

Open the AREA51Z.MCD file and click on one of the images on the middle. It will display all the images on PCK file.

Good night and thanks for the assistance :)
Title: Re: PCK View issues
Post by: davide on June 25, 2014, 08:17:23 am
Yesterday I did not copy files into MCDEdit folder.

Starting MCD file contains 137 tiles instead starting pck has 141 tiles

is it correct ?


Title: Re: PCK View issues
Post by: Hobbes on June 25, 2014, 11:02:43 am
Yesterday I did not copy files into MCDEdit folder.

Starting MCD file contains 137 tiles instead starting pck has 141 tiles

is it correct ?

Yes. Tiles can use more than 1 image if they are animated. In that case, the 4 extra images on the PCK files are used for the sliding doors animations.
Title: Re: PCK View issues
Post by: davide on June 25, 2014, 11:54:01 am
Thanks

diuring lunch time I will check it
Title: Re: PCK View issues
Post by: pmprog on June 25, 2014, 12:20:48 pm
Which PCKView app are you running? If it's the one from my pack (hmm, my github doesn't show a binary build for PCKView)

Anyway, I could try and take a look if I get time

Edit: Also: https://openxcom.org/forum/index.php?topic=2355.msg23676#msg23676
Title: Re: PCK View issues
Post by: davide on June 25, 2014, 12:37:58 pm
Which PCKView app are you running? If it's the one from my pack (hmm, my github doesn't show a binary build for PCKView)


I get source on github
Title: Re: PCK View issues
Post by: Hobbes on June 25, 2014, 12:50:49 pm
Which PCKView app are you running? If it's the one from my pack (hmm, my github doesn't show a binary build for PCKView)

Anyway, I could try and take a look if I get time

Edit: Also: https://openxcom.org/forum/index.php?topic=2355.msg23676#msg23676

I got the version posted on that thread you mentioned.
Title: Re: PCK View issues
Post by: Hobbes on June 26, 2014, 09:18:20 pm
So guys, any ideas whether this issue is related to PCK file capability or a bug?
Title: Re: PCK View issues
Post by: davide on June 28, 2014, 03:34:15 pm
I apologize for the delay  :-[

I think that the problem is an overflow in .TAB files

It contains position in PCK  in which begin the tiles.

The index was momorized by 2 byte (unsigned short).
The max value allowed is 65536.

When the size of pck file grow over that dimesion,
or better when a new tile begin after that position,
the TAB file is corrupted because the index of the new tiled was saved with an overflow
therefore them point on previous tile image at strange, but constant, position.

There is not a solution because it is connect to file format
 (same problem than maps ...sigh  :'()

 I will add a block in PCKView  to limit max number of tiles to  avoid the someone  loose
time about it.

You, correctly just ipotized that problem was connected with max tile numbers.

TAB file for TFTD Unit have 4 byte slot index.
This option was asked by PCKView before saving: the second  option enable it.

I could will correct PCKView/Mapview to recognize the format  2/4 byte index by compare file size with first entry (the tiles count).

But
The MCDEdit sorce code are not public
and
I do not know if OX engine is ables to read them without  a fix.

We need a new file formats at list to break the size limits imposed by pc performance of 1994

backward compatibility is mandatory and
all tools and engine could be aligned to support both file formats: (map, pck, tab, mcd) versus (map2, pck2, tab2, mcd2)

An analisy of requirements it is desirable.
Title: Re: PCK View issues
Post by: Hobbes on June 28, 2014, 08:40:25 pm
OK, thanks for all the work, it's nice to know exactly what is happening rather than personal conjecture.

Quite honestly, I don't think that this is a big deal since this only happened because I'm pushing the limits of the MCD/PCK/TAB files with a terrain I am redesigning and I was expecting that something like this might appear.

The best way to avoid this issue is to do what already happens with all terrains: split the tiles between different tilesets so that each individual TAB file does not go over the limit.

However the reason why the TAB file went over the limit is related to the MapView editor, since it has a limit of 256 tiles that can be used to design maps. If you use more than 256 tiles, then the reference numbers are reset from 0 and the map gets wrong when saved since it will contain the indication of using tile #10 on location x,y,z instead of using tile #266, which was what I wanted.

The reason why my TAB file got so big was that I realized that some tiles are never used when designing maps, namely the tile replacements for walls, objects, etc., when they get hit by gunfire. And so, I design the last tileset so that those tile replacements have numbers above #256 they'll never be needed to use on MapView, and I can use more than 256 tiles on OpenXcom, although I'll only use the first 256 tiles while designing maps with MapView. So I started to move those tile replacements to the end of the file and add new tiles instead but those but when I reached the 64kb limit that's when the graphic problems started.

Like I said, this isn't a big issue, since there aren't much more tile replacements that I can move. I had already moved 30 out of 40 to the end of the files before the graphic issues started, so the possible gains from changing MapView/PCKVIew/MCDEdit are not worth the effort for this case.

But thanks a lot for the assistance, it gets a lot easier (and less frustrating) when you realize that it is an actual limit derived from the file format rather than you're doing something wrong. :)
Title: Re: PCK View issues
Post by: TheBigSot on July 17, 2014, 06:08:21 am
Ok question:

1- The maximun limit Tiles allowed for UFO is 65536, and MapView has a bug that limits it to 256?
Title: Re: PCK View issues
Post by: davide on July 17, 2014, 07:34:29 am
The limit was into file format
Title: Re: PCK View issues
Post by: Hobbes on July 17, 2014, 02:41:36 pm
Ok question:

1- The maximun limit Tiles allowed for UFO is 65536, and MapView has a bug that limits it to 256?

The terrain limits for the original game are as follows:
* There's a maximum of 10000 squares (e.g. 50x50x4) on its total size when generating the battlescape. On OpenXCom this limit has been removed.
* Each terrain can have 256 tiles (the individual ground/walls/objects used on the squares) assigned. On OpenXCom this limit has been increased, but the .MAP file format still has the 256 tile limit. 
Title: Re: PCK View issues
Post by: robin on July 17, 2014, 03:01:18 pm
the .MAP file format still has the 256 tile limit.
Does that mean the sum of all the tiles from all the terrains associated to the map, or only the tiles actually *placed* on the map (so you can have a map with 256 terrains, using only 1 tile from each of them)?
Title: Re: PCK View issues
Post by: Hobbes on July 17, 2014, 03:29:16 pm
Does that mean the sum of all the tiles from all the terrains associated to the map, or only the tiles actually *placed* on the map (so you can have a map with 256 terrains, using only 1 tile from each of them)?

First a little clarification of my terminology, otherwise things get confusing :)
* A terrain is used to create a battlescape and is composed by both tilesets and maps (or map blocks)
* Tilesets are composed each of a MCD, PCK and TAB file. They contain 'tiles' or the components (ground, walls, objects) used on each map square. Usually there is more than 1 tileset associated with a terrain since the PCK files have a limit of 64 Kb.

So, regarding your question, the 256 limit applies to the sum of all tiles present on the tilesets associated with a terrain.
Title: Re: PCK View issues
Post by: davide on July 17, 2014, 03:44:06 pm
A Terrain is based on a "Data Sets"
A Data Set  is composed by concatenate a group of .PCK that contain the tile set

the map contain for each cell,  4  tile index of the current data Set.
Each index is memorized by a single byte and it is the limit.

as example the terrain URBAN used by TERROR_MISSION is defined by a data set composed by 5 file .PCK
The maps of this terrain contain index of the sum of tiles contained into the referred PCKs.
If you change order of PCK, or add one tile into a PCK of the current Data set,
 the index change too and your maps result corrupted.
therefor If two maps are designed with different Data Set , them are incompatible.

Code: [Select]

terrain:
- name: URBAN
    mapDataSets:
      - BLANKS
      - ROADS
      - URBITS
      - URBAN
      - FRNITURE


Title: Re: PCK View issues
Post by: robin on July 17, 2014, 04:06:20 pm
I knew about the "strict order" thing. Anyway thank you both for the clarifications (and sorry for the derailment, since the topic was about PCKView).
Title: Re: PCK View issues
Post by: moriarty on July 17, 2014, 04:10:12 pm
kind of off-topic, but does any of you have an idea why opening INTICON.PCK only shows me the first five icons? same with BASEBITS.PCK, for that matter. I'd really like to extract the craft weapon icons from both files, but I'm stuck.
Title: Re: PCK View issues
Post by: davide on July 17, 2014, 04:21:27 pm
Do you hane try with MCDEdit or BBTools ?
Title: Re: PCK View issues
Post by: moriarty on July 17, 2014, 04:25:49 pm
No... I guess I should. Do the BBTools extract any pck image?
Title: Re: PCK View issues
Post by: Falko on July 17, 2014, 04:33:39 pm
here
Title: Re: PCK View issues
Post by: moriarty on July 17, 2014, 05:14:28 pm
thank you!
Title: Re: PCK View issues
Post by: TheBigSot on August 03, 2014, 08:26:14 pm
The terrain limits for the original game are as follows:
* There's a maximum of 10000 squares (e.g. 50x50x4) on its total size when generating the battlescape. On OpenXCom this limit has been removed.
* Each terrain can have 256 tiles (the individual ground/walls/objects used on the squares) assigned. On OpenXCom this limit has been increased, but the .MAP file format still has the 256 tile limit.

Thanks for the info, I will review if possible
Title: Re: PCK View issues
Post by: Aldorn on August 03, 2014, 09:34:14 pm
here
Falko, is it possible for you to upload the BIGOBS, FLOOROB and HANDOB files too, I mean with the adequate palette ?
Title: Re: PCK View issues
Post by: Falko on August 03, 2014, 09:49:47 pm
done
Title: Re: PCK View issues
Post by: Aldorn on August 03, 2014, 09:58:25 pm
Thanks !!!! :)
(the 4 ! are for the 4 seconds it took you to answer me)
Title: Re: PCK View issues
Post by: TheBigSot on August 04, 2014, 12:12:11 am
I posted an upgrade to MapView integrated with PckView.
PckView should have some bugs fixed in that version

https://openxcom.org/forum/index.php?topic=1321.45