Playing with my new Android Smart Phone for 3 months, from rooting, to ROM flashing, then S-OFF, I have roughly came out with some basic understanding on how Android really works. Following are based on my experiences, some might based on some references, and might have some misunderstanding. Anything not right feel free to comment.
Android is Linux
This might be obvious but I point this out any way. So if you browse the file system using any file manager, you can see the file structure is following Linux style: /system, /data, /cache, /mnt and some more. Since it is Linux, Android is using EXT as hard disk format (Froyo using EXT3 while Gingerbread using EXT4).
ROM, Internal Memory, Memory and SD Card
We always heard of flashing ROM in Android, which the term is not exactly correct. Most of the time we are actually flashing the internal memory. Like a PC, Android boots from HBOOT (equivalent to BIOS) that stay at ROM, then loads the system files from Internal Memory, and lastly loads any applications that installed to SD Card.
Not sure what kind of format is used in ROM, but roughly it is partitioned into few partitions like HBOOT, RECOVERY and SPLASH1
Internal Memory is definitely in EXT format that partitioned into few partitions as well. The main partitions are SYSTEM (basically where Android files stay), CACHE and DATA (where application installed). So if you check the Internal Memory size from Settings, you might be confused by the size of it (much lower than the specification). Actually the size showed is the size of DATA partition, while the rest of the size is used by other partitions.
Lastly SD Card is in FAT format.
Rooting, Recovering and Flashing
Rooting in Android is more than gaining back the root privilege. In order to protect the phone from user, manufacturer puts a lot of security features to the ROM and Internal Memory. So rooting is actually mean disable the security features.
Aside from disable the security, rooting also includes step to install a recover tools in ROM. By using the recover tools, user can flash the internal memory with other version of Android (just boot into HBOOT and select to boot from recover, like multi boot in PC).
Unrevoked provides tool to do rooting for HTC.
S-OFF and Repartition
However flashing the internal memory only put in the new system files, but it cannot repartition it (you might want to have a bigger size of DATA partition to install more application). To do repartition, we need to do S-OFF and patch HBOOT. After S-OFF and the patching, we now can have the options to repartition by flashing HBOOT in ROM.
AlphaRev provides tool to do S-OFF and repartition for HTC.