Re: apex-1.3.31 and sercomm flash header
On Sun, Aug 06, 2006 at 02:34:06PM +0930, Rod Whitby wrote:
> Marc Singer wrote:
> > I'll tell you what I think we should do, then. I'll look at the FIS
> > format to make sure it makes sense. Let's add (at least) a descriptor
> > to the FIS record which is a pair (offset length). This will
> > communicate a span of bytes to omit in the partition. This has the
> > benefit of allowing a seamless integration of this 'skip' feature. In
> > APEX, the FIS driver will notice the skip information and bypass those
> > bytes without fuss. The UI will be the same to the user of APEX. All
> > we have to do is add the record to the FIS partition where the skip
> > occurs and then we have to insert those bytes into the partition data
> > when we write it.
>
> OK, so you've gone to the opposite idea of what I was thinking. You're
> suggesting we tell apex what to skip out of each existing logical
> partition, rather than telling apex to move multiple physical partitions
> to create a single logical partition at the destination.
I don't think that this is 'opposite'. I am interested in making a
change that is clean from the different perspectives, user, boot
loader, slugimage, and so on.
> So there are two alternate approaches currently proposed (see the end of
> this message for a third proposal):
>
> a) adding a single offset field in the FIS directory entry which tells
> apex to start the copy at that offset from the start of the FIS
> partition. Defaulting to zero retains the current behaviour. The value
> would need to be other than all zeros or all FFs to be considered valid,
> or we could go the whole descriptor and magic number deal here too.
>
> or
>
> b) adding support for skip descriptors in each partition which tells
> apex the location and size of holes in the partition data which need to
> be skipped over when copying the data.
>
> > Here's what I suggest. We can put these skip descriptors into the
> > partition table.
> >
> > struct fis_descriptor_skip {
> > char magic[4]; // Letter 's' 'k' 'i' 'p'
> > unsigned long offset; // Offset of skip from partition start
> > unsigned long size; // Count of bytes in the skip
> > }
>
> If we decide to go with (b), then that format seems fine to me.
>
> For (a) we could go with (data_offset is the new field):
>
> struct fis_image_desc {
> unsigned char name[16]; // Null terminated name
> unsigned long flash_base; // Address within FLASH of image
> unsigned long mem_base; // Address in memory where it executes
> unsigned long size; // Length of image
> unsigned long entry_point; // Execution entry point
> unsigned long data_length; // Length of actual data
> unsigned char _pad[256-(16+8*sizeof(unsigned long))];
> unsigned long data_offset; // Offset of actual data
> unsigned long desc_cksum; // Checksum over image descriptor
> unsigned long file_cksum; // Checksum over image data
> };
>
> or do a similar descriptor scheme to option (b).
>
> > We can put this anywhere in the _pad area of the fis_descriptor. I
> > recommend that we put it at the end so there is little chance of it
> > colliding with any possible changes to the FIS structure. Moreover,
> > it doesn't need to be at any particular place in the padding region.
> > APEX will scan the whole area for the magic numbers. There will be as
> > many skip regions as there are fis_descriptor_skip's found.
>
> Yep, I agree that any new apex-specific descriptors should go at the end
> of the padding if they are going in the FIS directory. See option (c)
> below for an alternative place to put offset information.
>
> For options (a) and (b) we would need to do some homework to see if the
> CRC fields (which aren't used by the nslu2, but may be used by redboot
> on other devices like the nas100d - I need to check) need to be updated
> to include the new apex descriptors or not.
I agre that we'd need to do some homework for b), but I think that a)
can be done without any changes to the partition table
> > If you like this idea, I can implement the feature in APEX without
> > much hassle. All we'd need is agreement on the format and a patch to
> > slugimage to cope with the new feature.
>
> Let's discuss the pros and cons of the two approaches, and then I will
> propose a third.
>
> The two options are equally powerful as far as being able to copy any
> number of any sized pieces anywhere, overlapping or abutting as
> required. So that's not a deciding factor.
>
> In both cases, the image creation/flashing tool (slugimage or upslug2)
> needs to know how to split the kernel into two pieces, inserting a 16
> byte dead spot in the middle, and then give instructions to apex on how
> to deal with it. For option (a), the instructions become part of the
> boot commands (i.e. you need to load an extra partition). For option
> (b), the boot commands are the same as the before (so the apex user
> documentation and work-flow doesn't need to change), the extra
> information is hidden in the partition table, and the user doesn't
> really need to know about it at all (they can treat the kernel partition
> as they did before - as far as they know it has just increased in size
> by 16 bytes).
>
> One downside with option (b) is that the stuff in the middle of the
> kernel partition is "hidden" from the user, so someone accessing that
> mtd partition from Linux gets a corrupted kernel if they copy it out.
> Silently pulling stuff out of the middle of something just seems to me
> to be a recipe for trouble and confusion.
>
> With option (a), they get two pieces of a kernel (which should be
> obvious by the partition names, like kernel1 and kernel2), each of which
> has a 16 byte header (which is also reasonably obvious).
We don't need to make any changes to make this happen. APEX already
has a simple way to start copying part way into a region--regions are
an APEX concept that describe a data source or data sink.
apex> copy fis://kernel+16 $bootaddr
This will copy the kernel image to $bootaddr, skipping the first 16
bytes. This is how it is done right now.
> I guess we could also do an nslu2-specific patch to the mtd driver to
> understand the skip descriptor, but that might be taking things a bit
> too far and it would never be accepted upstream.
I don't think that this would be necessary. Remember, people cannot
write kernel images without inserting SERCOMM headers. We'd just
require that one is inserted into the middle of the kernel partition.
User's will always have to know to remove these headers.
>
> Alternative (c):
> ----------------
>
> A third alternative (let's call it (c)) would be to use the 12 unused
> bytes in the SerComm length header for magic and data offset fields. So
> the 16 bytes prepended to the two parts of the kernel would be:
Are you sure that the SERCOMM header is really just four length bytes
and padding? I thought that they defined all of the bytes for one
purpose or another.
> struct fis_partition_header {
> unsigned long size; // Unchanged, currently used by SerComm
> char magic[4]; // Some magic string ('apex'?)
> unsigned long offset; // Offset of data from partition start
> // - would be set to 16 in our case.
> char padding[4]; // Padding, might be used in the future.
> }
>
> Then the FIS table format and contents don't need to change at all
> (apart from adding an addition entry in the FIS directory), and we don't
> have any potential problems with CRC entries, or differences in sizes
> before and after the copy.
>
> Apex just needs to look for this magic header on any partitions loaded
> (and the set of magic strings could be expanded to do more things than
> just an offset at the start), and the code which prepends the header
> field to the kernel before flashing just needs to split the kernel into
> two pieces for two partitions and prepend the correct headers.
>
> No change to the FIS directory format is required and MTD partitions are
> unchanged in format from today (some already have these headers prepended).
>
> You just need two apex commands (instead of one) to load both halves of
> the kernel to abutting locations. You might even find that the kernel
> splits nicely into kernel (plus some optional padding) and initramfs for
> the two partitions ...
>
> What do you think about this third alternative?
>
Considering that you don't like b), I prefer a). I'll elaborate.
In all of these cases, the data written has to be organized in a
manner peculiar to the slug. In all cases, the kernel is split in two
and cannot just be read or written without knowledge of the custom
format. Remember that the kernel is also byte-swapped in Debian
Linux. The only substantive difference between a) and b) is that
there are two partitions in a) and one in b). Hence, I disagree with
your downside to b) argument. No matter how people use our special
slug loader, they will have to insert 16 bytes at the appropriate
place in the middle *and* that have to know how to do the necessary
byte swapping.
I prefer b) because it has the simplest implementation and use.
1) There is a localized change to APEX's fis driver where it scans
the pad area for skip headers. The command to copy the kernel is
unaffected.
2) Slugimage is modified to accept a new kind of 'file' that has a
fixed offset in the ramdisk image with an extra addition to the
partition table of the affected partition to indicate that this
is 'skipped'.
(BTW, it hadn't occurred to me to use a SKIP descriptor to skip the
first 16 bytes of a partition. I was only concerned about locating
the buried SERCOMM header in the middle of the kernel partition.
We could define two skip headers, but that isn't really necessary
or desirable as the header at the start of the partition contains a
valid sercomm header.)
The changes for a) would be:
1) Addition of an APEX startup command to copy the two pieces of the
kernel. This will require a small bit of hard coding of the
second load address since APEX only knows about the kernel start
address. Note that this isn't a code change to APEX, just a
configuration change.
2) Slugimage is modified to split the kernel into two partition and
insert the 16 byte sercom header in the middle. I'm not sure how
you would do this, but I imagine that the easiest method would be
to tell it to put the kernel file in the KERNEL1 partition and
the overflow into KERNEL2. You'd still have to tell it about the
data to insert at the start of the second parititon.
I'm not in favor of c), but perhaps I don't yet understand it. There
isn't any reason to encode the offset from the start of a partition
since it is already easy to do in the copy command. Moreover, I don't
like hiding data like this in a boot-loader specific manner. The
mechanism in b) isn't specific to APEX. It's similar to a bad-block
table that any application would need to honor.
Option a) (without modifying the partition table) is about as simple
as we're going to get. The hardest piece is modifying slugimage to
put the pieces in the right place, and I don't expect this to be much
of a challenge.
Cheers.
Reply to: