Home Page The Club Computers News Links Glossary EYAWTK

Everything You Always Wanted To Know
But Were Too Polite To Ask

Amiga Technical References

Beyond the Four Gigabyte Barrier

Author: dhomas trenn Published by: NewTekniques magazine (US) Date: October 1998

UPDATE: December 19, 1999.

About OS 3.5

Amiga has released OS 3.5 and added official support for hard-drives larger than 4 Gigabyte. The previously released New Style Device (NSD), as discussed below, is now the official supported standard.

It is important to realize that the added NSD support does NOT mean that software that uses a custom file system, such as Movie Shop, will now work with larger hard-drives. If this software didn't work without NSD, it still won't work with it.

Also, hard drive applications such as DiskSalv, Ami-Back Tools, etc. will still not work properly, unless they are updated to support NSD.

OS 3.5 includes updated versions of Info, Format, Diskcopy, FastFileSystem and HDToolBox.

Greater Than 4 Gigabyte For FREE

The FastFileSystem included with OS 3.5 has long been available on the Amiga website (see below) for FREE. The problem is that the various versions of the FastFileSystem available there include expiry dates (which are now past). This means that each time you reboot your Amiga, a yellow alert will warn you of expiration. It is important to know that it is JUST A WARNING, and it can be ignored without any worry of problems with your hard drives.

If you find the alert to be too annoying, there is a patch available on Aminet, disk/misc/FFS43.19_patch.lha, that will disable the warning in the 43.19 FastFileSystem. It should ONLY be used with the 43.19 version though.

Alternatively, if you own OS3.5 but just want to update the hard drive support, you can use the 45.1 version from OS 3.5 with OS 3.1, without any problems.

Versions of Info and Format, that support greater than 4 Gigabyte drives, are available on Aminet for FREE - and it is still safe to use the older HDToolBox (see below).



Some time ago, I purchased two Quantum Fireball 4.3 Gigabyte (GB) hard disk drives. One was intended for AmigaDOS storage and the other for use with MovieShop and the V-Lab Motion video card. What I thought would be a simple and somewhat ordinary task of setting up a couple hard disks, turned out to be a great nightmare of confusion and misinformation.

There has been a great deal of discussion about this topic in newsgroups and emailing lists lately. Unfortunately, there is also a huge amount of misinformation floating around - which, I suspect, has confused some people.

If you are planning on using a hard disk larger than 4 Gigabytes with the Amiga, whether it be for video purposes or other, there are a few things you need to be aware of. Note that applications that bypass the Amiga's file system may not be affected by this problem.

The First Signs of Trouble

I set out to configure one drive as a single 4.3 Gigabyte AmigaDOS partition using HDToolBox. By default HDToolBox sets up two partitions, so I deleted the second partition and proceeded to maximize the first one. I dragged the little pointer all the way to the end, only to discover that my 4.3 Gigabyte hard disk had suddenly shrunken to only 14 Megabytes (Meg) - or so HDToolBox was telling me. I slid the slider back a bit and watched carefully as I increased it one position at a time... 4093 Meg, 4094, 1 Meg, 4, 7, 10, 13, 14 Meg. Something was seriously wrong.

As a computer programmer, I recognized what the error was right away; but, that did not mean that I immediately knew how to solve the problem.

What is Really Going On?

It is not too difficult to understand what causes the 4+ GB problems, though perhaps a little bit technical (bare with me for a minute). Simply put, AmigaDOS uses 32-bit pointers to reference positions on a hard disk and therefore can reference a maximum of 2^32 bytes or 4,294,967,296 (4 GB) - Note that, with some older SCSI controllers and devices the limitation is actually 2 Gigabytes. Many years ago, nobody expected that we would want to be using storage media as large as we do today. Understandable perhaps, consider that my first computer, a Commodore PET 2001, only had 8,192 BYTES of RAM. That was a lot back then!

If a program attempts to read or write data to an area of the hard disk above the 4 Gigabyte limitation, the data pointer will 'wrap around' (4,294,967,295 + 1 = 0, + 2 = 1, etc.). If this happens while the computer is writing data, existing information on the lower part of the hard disk will be overwritten (See Figure 1). This means that data will be lost and very possibly the Rigid Disk Block (RDB), which among other things stores the partitioning information for the hard disk, will be corrupted.

Figure 1

Figure 1

Unfortunately, all of this happens quietly and without any warning. You may not even know anything is wrong until it is too late. In particular, the Amiga generally only determines that a hard disk is 'invalid' when it is booting, otherwise it tends to assume that everything is normal. So, unless you tried to access any of the corrupted data, you would not be aware that there was a problem. But, when you rebooted the computer you would quickly discover that the Amiga no longer recognized the hard disk as valid. If you are really lucky, restoring the RDB would allow you to access some data; but, it is not very likely.

It gets a little more confusing when you try to determine exactly where the barrier exists; because a Gigabyte is not always a Gigabyte. Have you ever wondered why, for example, a 730 Meg hard disk, after formatting, has less than 730 Megabytes of storage? This is partly because hard disk manufacturers, when it comes to specifications, speak their own language. They refer to one Megabyte as 1,000,000 bytes, but any computer programmer will tell you that one Megabyte is equal to 1024 Kilobytes or 1,048,576 bytes, which is what the Amiga thinks, too. One can only guess what they might be referring to when they say one GB. Gigabyte envy??

The Easy Answer

To avoid the problem, the easiest thing to do is to ignore everything above the 4 Gigabyte barrier. It is not very elegant, but it does work 100% without any system patches or restrictions. This might not be a particularly great solution for larger drives, but for a 4.3 Gigabyte drive, you only lose about 150 Megabytes of storage.

It is necessary that you set the lower partition to no bigger than 4095 Meg. To be on the safe side with AmigaDOS, I recommend 3980 Meg (See Figure 2). For a 4.3 Gigabyte drive, that would leave 151 Megabytes for the last partition. Name it DEAD0 or something similarly obvious and never format it. You can further divide the lower partition into several smaller parts if you wish, but make sure to not go above 4095 Meg (or 3980 Meg if you want to be really safe). So that you do not see an icon for DEAD0, make sure that the 'Automount this partition' option is NOT checked for that partition.

Figure 2

Figure 2 - Partition Drive
(Note: this graphic has been altered to fit within available space).

The Solutions

I am not convinced that there is one do-it-all solution. I am also not yet sure that any of the following options work 100%. However, after talking with others, discussing this with the appropriate programmers, and through much trial and error I have come up with some information that will hopefully help you through this Giga-craziness.

The most important thing, and I can not stress it enough, is that you should backup ALL your hard disks before you start playing around with any of this. Technically, the only drive of concern is the one you attempt to use these patches with. However, as I have learned too many times, when things get really frustrating it is easy to accidently re-format the wrong hard disk. In seconds, it will be too late.

Consult each program's documentation for detailed information on supported controllers, setup and usage.

New Style Device - NSD

The official solution is from Amiga International Inc. and it is the method that will most likely remain compatible with future updates to the operating system. So, if you can get it to work, it is the best solution. This method is based on a device handling system called New Style Device that was originally developed for the Walker. NSD is not only a fix for the 4 GB problem, but can also be used to improve performance of other hardware, among these: sana2, serial and parallel devices.

There are three parts: an updated FastFileSystem (FFS43_19.lha), a scsi.device patch (SCSI_IDE43_23.lha) and the NSDPatch (NSDPatch43_20.lha ) program which adds support for many third party hard disk controllers.

Currently, NSDPatch includes support for the following controllers and trackdisk devices: Amiga's scsi.device and 2nd.scsi.device; Phase 5's 1230scsi.device, 1260scsi.device, 2060scsi.device and cybscsi.device; Draco's dracoscsi.device; HiSoft's squirrelscsi.device; Oliver Kastl's atapi.device; WarpEngine's warpdrive.device; DataFlyer's ExpSys.device; Hardital Synthesis' syndisk.device; BSC's oktagon.device; Ralph Babel's GuruRom omniscsi.device; Microbotics' HardFrame.device; and the diskspare.device. Other devices may also be configurable.

First, it is necessary to install the new file system using HDToolBox. In the main window, select the appropriate drive and then the 'Partition Drive' gadget to get to the 'Partitioning' section. By check-marking the 'Advanced Options' gadget (See Figure 2), select the 'Add/Update...' gadget to get to the 'File System Maintenance' window (See Figure 3). If another file system is already listed you should delete it first using 'Delete File System'. Select the 'Add New File System...' gadget to install the new file system.

Figure 3

Figure 3 - Add/Update File System (NSD)

Then, if you are using the scsi.device, you should install the SCSI_IDE43_23 patch appropriate for your machine. Otherwise, you should install NSDPatch. The current version of NSDPatch will not automatically activate partitions above 4 GB. You have to specifically tell it to do so in the NSDPatch.cfg file. This is expected to be changed in a future release.

I have been using this file system via the NSDPatch with Phase 5's CyberSCSI-II module for over a year now, without any problems.

You can get this free software from the Amiga International files area: http://www.amiga.de/files/index.html. At this stage, the software is still beta; but, it is very stable. These files are updated occasionally, so keep checking back for newer versions. These programs require OS 3.1.

Track Disk 64 - TD64

TD64 is a file system replacement (patch) written by Ralph Schmidt (Phase 5) based on the Track Disk 64 standard, which has been defined by an independent group of third-party Amiga developers. It also requires OS 3.1.

Currently, support is included for: Phase 5's 1230scsi.device, 1260scsi.device, 2060scsi.device, Fastlane z3scsi.device and cybscsi.device; BSC's ALF.device and oktagon.device; Draco's dracoscsi.device; Ralph Babel's GuruRom omniscsi.device; trackdisk.device and the carddisk.device. The following devices are not compatible: All Amiga SCSI devices (including scsi.device, 2nd.scsi.device; and hddisk.device); GVP's scsidev.device; and Microbotics' HardFrame.device.

This file system is installed using HDToolBox, in the same way as described above for the NSD file system.

You can get this free software from Aminet: disk/misc/ffstd64.lha

Figure 4

Figure 4 - Add/Update File System (TD64)

Bigdisk

Bigdisk is a commercial program written by Janos Farkas. It was designed for use with MovieShop (V-Lab Motion) and is currently the only known solution to work with it. However, it can also be used with the Draco and AmigaDOS 2.0 or greater. It has not been tested, but it may even work with earlier versions of AmigaDOS.

Bigdisk approaches the 4 Gigabyte barrier in a different way, taking advantage of the SCSI command set to access the full capacity of a hard disk. When you start the Bigdisk initializer (Biginit), it overlays a new bigdisk.device over the existing device (scsi.device or other) and divides the destined hard disk into 4 Gigabyte sections, creating several virtual drives. Bigdisk 'Unit 0' refers to the first 4 Gigabytes of the original device, 'Unit 1' to the second 4 GB, and so on. Requests sent through Bigdisk are redirected and passed on to the underlying device which can then handle them properly (See Figure 5).

Figure 5

Figure 5 - Bigdisk Virtual Devices

Unlike with NSD and TD64, Bigdisk can not be used to create a single partition bigger than 4 Gigabytes.

The program has been tested and is known to work with the Amiga's scsi.device and z3scsi.device; and the Draco's dracoscsi.device. At the time of writing, it did not work with Phase 5's cybscsi.device. Other devices may or may not work. Fortunately, a free demo is available which allows you to test the program with your system.

To get the demo, ordering information or other details, visit the Arthur Wilkins Software website at: http://www.acomp.hu/awsw/

Concerns, Cautions and Important Notes Conclusion

When it comes to using hard-drives larger than 4 Gigabyte, the Amiga does have some problems. But the solutions described here, in particular NSD, are safe and very reliable.


Copyright 2000 Amiga Auckland Inc. All rights reserved.
Revised: June 16, 2005.