Local variables using other structs and being unable to access type not in memory

[AndTheDaysGoBy]

So, I'm experiencing somewhat confusing. The premise is this:
if field a is resolved before field b, you would think that you can reference the children of field a in b without issue. However, I'm experiencing issues, but only with my real data.

In the test I made, thinking that maybe it was a bug, I did NOT experience an error.

struct CArchive {
    u32 magic;
    u32 entries[parent.count];
};


struct Attribute {
    u32 current_index = std::core::array_index();

    u32 value = parent.child.entries[current_index] [[export]];
};


struct Archive {
    u32 magic;
    u32 count;
    CArchive child @ (addressof(this) + 14);

    Attribute attributes[count];
};

In this, it just works, attributes, which is referenced AFTER child, properly is able to access child's entries.

However, when attempting to do so with my real data, I run into:
E: runtime error: Cannot access data of type that hasn't been placed in memory.

Here is a more similar example to what I actually have which DOES reproduce the error, but I don't actually see how it differs from the one that works, nor why an error is occurring.

import std.core;


struct NestedEntry {
    u8 length;
    u8 name[length]; // (only second pass through) E: runtime error: Cannot access data of type that hasn't been placed in memory.
};


struct NestedArchive {
    NestedEntry entries[parent.count];
};

struct Attribute {
    u32 current_index = std::core::array_index();
    NestedEntry entry = parent.parent.archive.entries[current_index]; // cause of issue as it tries to re-process NestedEntry
};


struct Entry {
    char name[];
    Attribute attributes[parent.count];    
};

struct MyArchive {
   // Broadly speaking, it looks like this.
    u8 version;
    u8 count;
    u16 data_offset;

    NestedArchive archive @(addressof(this) + data_offset);
  
    Entry entries[count];
};

MyArchive archive @ 0x0;

Have a file that starts with 16 bytes, with the first 3 bytes as 01 02 0A.

[paxcut]

The problem is probably with the assignment of structures which can fail if the structures are complex. The thing is that you rarely need to assign structs by value when you can just create pointers to reference them instead, so maybe try

NestedEntry entry @ addressof( parent.parent.archive.entries[current_index]); // cause of issue as it tries to re-process NestedEntry

[paxcut]

struct C {
     B b @ addressof(parent.b); /*
          Will error now since this doesn't just "point" to the already processed B, but rather seems to re-execute
          the logic in the struct, re "instantiating" B. This re-executed logic has B attempt to resolve `parent.count`
         but parent in this context is `C`, and `C` has no variable `count`.
    */
};

here parent means parent of C not parent of B. parent refers to the structure where it is used, not to the member where it is being used

[paxcut]

Ah, you mean the parent inside B called when creating B. you can also add a count to C using u8 count = parent.count;

[paxcut]

Also note that using assignment would have created the same error as well since you are creating a copy of B inside C.

[AndTheDaysGoBy]

You're right, sorry, but hopefully my example still makes sense to you. Just add one more layer of depth with struct C being wrapped in a struct D. The point I'm getting at is that Thing thing @ addressof causes a reprocessing of a piece of file data into a struct. It does not just grant you a pointer to an already processed struct. This distinction can lead to issues since some child structs leverage parent, parent.parent, etc. which makes the resultant filled out struct context dependent.

So, the hope would be to have a way to have a struct be populated with data once, and then have a way to access that populated/post-processed struct itself, thereby avoiding the re-execution of the logic in the struct's definition. Maybe that's not possible. E.g.

import std.core;


struct NestedEntry {
    u32 current_index = std::core::array_index();

    u8 length;
    u8 name[length];
    if (current_index == parent.count - 1)  {
         // NestedEntry, when instantiated as part of NestedArchive will work due to NestedArchive.count
        // but, since addressof causes re-caculation of the struct, when referenced in Attribute, it will fail.
        // Hence the desire to have a way to reference the already processed struct.
         u8 terminal_byte;
    }};


struct NestedArchive {
    u8 count;
    NestedEntry entries[count];
};

struct Attribute {
    u32 current_index = std::core::array_index();
    NestedEntry entry = parent.parent.archive.entries[current_index]; // cause of issue as it tries to re-process NestedEntry
};


struct Entry {
    char name[];
    Attribute attributes[parent.count];    
};

struct MyArchive {
   // Broadly speaking, it looks like this.
    u8 version;
    u8 count;
    u16 data_offset;

    NestedArchive archive @(addressof(this) + data_offset);
  
    Entry entries[count];
};

MyArchive archive @ 0x0;

Edit: I see your new comments. I agree that one could add count as a local variable to Attribute, and I agree that that's a way to get by this. But then that makes the struct's context pathological, infecting anyone using it. That's why, if there is a way to reference a post-processed struct, it would be very useful. Does that already exist? Could it be a feature request?

[paxcut]

Maybe the problem is because Nested_Archive has a count variable separate from the one in MyArchive so if they are not equal you will access values that don't exist.

[paxcut]

you have MyArchive.count entries and each has MyArchive.count attributes. So you are creating count squared references to structures of which there are only Nested_Entries.count of them. How does that work?

[paxcut]

But then that makes the struct's context pathological, infecting anyone using it.

Actually the suggestion of adding count to Attributes was a bad one. It is bad because it is based on a bad format spec design. There should be only one source for the number of elements in the array that is to be referenced. Having count in both MyArchive and NestedArchive can lead to the creation of null references so the correct suggestion should have been to make NestedEntry use parent.parent.count. The following code follows that suggestion and worked on the sample input provided

import std.core;
// I tested this pattern with the following hex values
#pragma example 00 02 10 00 74 65 73 74 00 6E 61 6D 65 00 00 00 02 61 62 03 61 62 63 00 

// this pattern is dynamic and its size depends on where it is placed.
// I suspect it cannot be used for local variable assignments
// because it needs to be instantiated before assignment.
struct NestedEntry {
    u32 current_index = std::core::array_index();
    u8 length;
    char name[length];
    if (current_index == parent.parent.count - 1)  {
         // if NestedArchive.count and MyArchive.count are different the references would break
         // so NestedArchive should not define count and use its parent's instead which forces 
         // NestedEntry to use parent.parent. Therefore Entries is the only struct that needs 
         // to define count but it already uses it so it makes sense to add it there.
        
         u8 terminal_byte;
    }
};


struct NestedArchive {
    // count shouldn't be defined here and doesn't need to be.
    NestedEntry entries[parent.count];
};

struct Attribute {
    // this now works as expected. I'm not certain but the error seen when using = 
    // could be created if pattern needs to be  instantiated prior to the assignment
    // due to its dynamic nature.
    NestedEntry entry @ addressof(parent.parent.archive.entries[std::core::array_index()]); 
};


struct Entry {
    char name[];
    // this is the only place count needs to be added but seems like it can use it too
    u8 count = parent.count;
    Attribute attributes[count];    
};

struct MyArchive {
   // Broadly speaking, it looks like this.
    u8 version;
    u8 count;
    u16 data_offset;

    NestedArchive archive @(addressof(this) + data_offset);
  // having count entries seems wrong too. there is only one array of size count
  // already defined in Entry
    Entry entries[count];
};

MyArchive archive @ 0x0;

[AndTheDaysGoBy]

count does have to be defined in the NestedArchive though. Remember, at its core, they are separate archives. E.g. for lack of a better example, say Archive is an archive of textures, which contains a child archive with metadata.
The parent archive, Archive, might have 20 textures, so count == 20, whereas the child archive might only have a count of 3 if it's a reference of fields, position, velocity, and acceleration.

This misunderstanding stems from me though. In my example above, I had

struct Entry {
    char name[];
    Attribute attributes[parent.count];    
};

when I should've written

struct Entry {
    char name[];
    Attribute attributes[parent.archive.count]; 
};

i.e. fully. with data

01 02 0A 00 54 31 00 54 32 00 02 02 49 44 03 50 4F 53 00 00 00 00 00 00 00 00 00 00 00 00 00 00

import std.core;

struct NestedEntry {
    u32 current_index = std::core::array_index();
    u8 length;
    char name[length];
    if (current_index == parent.parent.count - 1)  {
         u8 terminal_byte;
    }
};


struct NestedArchive {
    u8 count;

    // NestedArchive.count many NestedEntries in the NestedArchive, e.g. NestedArchive.count many fields (ID, Pos, etc.).
    NestedEntry entries[count];
};

struct Attribute {
    NestedEntry entry @ addressof(parent.parent.archive.entries[std::core::array_index()]); 
};


struct Entry {
    char name[];

   // Has parent.archive.count as the count, Since we're just re-positioning the nested Entries as attributes on the parent archive's entries.
    // E.g. if you comment out this attributes field, it should just work and you'll get a structure of
    /*
archive (Archive)
    |- version
    |- count
    |- entries
         |- [0]
              |- name: "T1\x0"
         |- [1]
              |- name: "T2\x0"
    |- archive (NestedArchive)
        |- count
        |- entries
             |- [0]
                  |- name: "ID\x0"
             |- [1]
                  |- name: "POS\x0"
    */
    // But what I want is for this attributes field to work so that I can have
   /*
    |- version
    |- count
    |- entries
         |- [0]
              |- name: "T1\x0"
              |- attributes
                   |- [0]
                       |- name "ID\x"
                   |- [1]
                       |- name "POS\x"
         |- [1]
              |- name: "T2\x0"
              |- attributes
                   |- [0]
                       |- name "ID\x"
                   |- [1]
                       |- name "POS\x"

    |- archive (NestedArchive) // I would mark this [[hidden]]
        |- count
        |- entries
             |- [0]
                  |- name: "ID\x0"
             |- [1]
                  |- name: "POS\x0"
   */
    Attribute attributes[parent.archive.count];
};

struct MyArchive {
   // Broadly speaking, it looks like this.
    u8 version;
    u8 count;
    u16 data_offset;

    NestedArchive archive @(addressof(this) + data_offset);
    // MyArchive.count entries on the top-archive, e.g. MyArchive.count many textures
    Entry entries[count];
};

MyArchive archive @ 0x0;

Thus, sizeof(Entry) * sizeof(Attribute) == sizeof(Entry) * sizeof(NestedEntry). Since the whole idea is to take the attributes in the nested archive and just re-position them in the parent archive.

I do want to, just in case say, please do not spend too much time trying to find a working solution for this. After all, then I'm just having you do my work lol, and to waste a contributor's time on my specific problem seems unethical. All I really need from you is to know if something is possible with the language which, if it isn't, I'll have to just give up on attempting to re-structure the data within imhex itself, and do that external to it. (Of course, I would then open a feature request for there to be a way to have a pointer to post-processed structs, but that could be either accepted or denied at your and the maintainer's choice).

[paxcut]

Ok, that makes more sense. I think the problem is that we don't really need to create the pointer to archive before entries since we can create the nested entries inside the entries. If this is what you are after:

this code will generate that

import std.core;

struct NestedEntry {
    u32 current_index = std::core::array_index();
    u8 length;
    char name[length];
    if (current_index == parent.count - 1)  {
         u8 terminal_byte;
    }
};


struct NestedArchive {
    u8 count[[hidden]];

    // NestedArchive.count many NestedEntries in the NestedArchive, e.g. NestedArchive.count many fields (ID, Pos, etc.).
    NestedEntry attributes[count];
}[[inline]];


struct Entry {
    char name[];

   // Has parent.archive.count as the count, Since we're just re-positioning the nested Entries as attributes on the parent archive's entries.
    // E.g. if you comment out this attributes field, it should just work and you'll get a structure of
    /*
archive (Archive)
    |- version
    |- count
    |- entries
         |- [0]
              |- name: "T1\x0"
         |- [1]
              |- name: "T2\x0"
    |- archive (NestedArchive)
        |- count
        |- entries
             |- [0]
                  |- name: "ID\x0"
             |- [1]
                  |- name: "POS\x0"
    */
    // But what I want is for this attributes field to work so that I can have
   /*
    |- version
    |- count
    |- entries
         |- [0]
              |- name: "T1\x0"
              |- attributes
                   |- [0]
                       |- name "Pos\x"
                   |- [1]
                       |- name "Pos\x"
         |- [1]
              |- name: "T2\x0"
              |- attributes
                   |- [0]
                       |- name "Pos\x"
                   |- [1]
                       |- name "Pos\x"

    |- archive (NestedArchive) // I would mark this [[hidden]]
        |- count
        |- entries
             |- [0]
                  |- name: "ID\x0"
             |- [1]
                  |- name: "POS\x0"
   */
   NestedArchive archive @ parent.nestedArchiveAddress;
};

struct MyArchive {
   // Broadly speaking, it looks like this.
    u8 version;
    u8 count;
    u16 data_offset;
    u64 nestedArchiveAddress = data_offset + addressof(this);
    Entry entries[count];
};

MyArchive archive @ 0x0;

This saves you from the whole ordeal of having to reposition the nested entries and also from having to hide the archive that you never use other than as attributes.

[paxcut]

A small improvement:

   NestedArchive archive @ parent.data_offset + addressof(parent.version);
};

struct MyArchive {
   // Broadly speaking, it looks like this.
    u8 version;
    u8 count;
    u16 data_offset;
    Entry entries[count];
};

[AndTheDaysGoBy]

Let me first preface by saying, what you posted WILL work.
My "but" I think I'd call this more so a... not theological, but, a perspective type of thing?
What I mean is, what is the "pattern" meant to represent? The reason I had the structs which actually read the data directly as they were is because, that structure mirrors exactly how it looks in the file. To me, that is the purpose of the pattern, I wanted to have a second layer to be able to change just the representation of that visually in the "Pattern Data" section.
In contrast, what you have is you directly fill the structs in the visual representation you desire. That is fine to do, it's just a difference in viewpoint of what the pattern file should represent.

E.g. in this context, representation wise, you have

entries
|- [0]
    |- archive
     ....
|- [1]
    |- archive
    ....
|- [2]
    |- archive
    ...

Because you aim to leverage the inversion, of referencing the archive in each entry, to ensure those nested attributes appear per entry. That makes sense, but it also now instantiates the archive three times, implying a representation that each entry has a copy of the archive, when isn't actually the case "in the file."

That was the idea behind pointers to post-processed structs, you keep the true representation of the file, while manipulating the visuals via local variable pointers.

|- child_archive
|- entries
    |- [0]
         |- name
         |- < some pointer to the parent's child archive stuff>
    |- [1]
         |- name
         |- < some pointer to the parent's child archive stuff>

Just so that I'm not crazy, do you get that idea of having two layers? I.e., fill the structs to mirror the file as is, and then a second layer that could use local variable pointers to re-arrange the representation of data. Does it sound like a good idea or a cool idea, or do you think the innate structure of the pattern should match the visual representation you want, regardless of the underlying file representation?

[paxcut]

We are deviating from the discussion topic. In order to make the discussion useful to others we should focus on resolving the issues as described on the OP. The discussion is about local variables of custom types giving errors when defined, but before that topic is stated you find

if field a is resolved before field b, you would think that you can reference the children of field a in b without issue.

there is a claim that there are problems referencing already existing variables so it becomes one of the discussion topics.
You posted two examples of patterns and then say:

However, when attempting to do so with my real data, I run into:
E: runtime error: Cannot access data of type that hasn't been placed in memory.
Here is a more similar example to what I actually have which DOES reproduce the error, but I don't actually see how it differs from the one that works, nor why an error is occurring.

So there are three questions that need answering.

1. Why can't the pattern reference an already created variable?
2. How are the two examples different?
3. why is the error occurring?

To answer the first one note that when you run the pattern with the error you get the following indicators for where the problem lies

To clarify,you are not creating a reference to the existing value. You are attempting to copy values from an existing struct to a local variable that is instancing the struct type. So instead of referencing we should say accessing the values of the existing variable.
The location of the errors clearly shows that there is no problem accessing the element of the existing variable and the problem is entirely in either the assignment of the values to the local variable or in the declaration of the variable due to its type. Further evidence of this is the fact that you can create actual references using calculated pointers which work when done properly, so the accessing works and the first question is answered.

For 2) the clear difference between the two examples is the type of the local variable created to receive the values of the existing variable. The example code that runs ok uses a type that has no problems creating an instance of the type that is not placed in memory. In this case an unsigned 32 bit integer. The example where the error occurs uses a type that cannot be instantiated unless it is placed in memory. This answers question number 2.

What's the error all about? Although it is somewhat cryptic the message is n accurate description of the issue. Tying it together with the stack trace shows exactly where the problem is. The image shown below is the result of running the example that has the errors but with the placing of the MyArchive pattern removed and attempting to define a local variable of type NestedEntry in the global scope.

.
.
.
// MyArchive archive @ 0x0;
NestedEntry ne;

It is clear that a local variable cannot be created because the pattern is unable to create the name char array. Any pattern that contains arrays cannot be instantiated to create local variables. The evidence is easy to find. remove the array specification but leave the char name variable. also create a global count variable

u32 count =2;

struct NestedEntry {
    u32 current_index = std::core::array_index();
   
    
    u8 length;
    char name;//[length];
    if (current_index == count - 1)
        u8 terminal_byte;
};

now the code just above this runs. Why does the error occur? simply put the pattern language evaluator cannot determine the size that the array should have. You can argue that we are copying the contents of an existing variable so it could use the size of the lhs to determine the size of the array, but assignment is an operator that takes two parameters and the parameters have to be fully instantiated variables of the same type. Those must exist before any member assignments can be made. This answers question number 3.

If the post seems too long it is because I wanted to answer all the issues raised in one go, but I am willing to discuss any points of contention that may arise as long as they are directly related to the discussion at hand like ways to get around the local variable declaration errors of this type or feature requests aimed to do the same.