Most people who have worked on other Unix flavors like Solaris, HP-UX, IBM AIX, etc not used to Linux may have never heard of an OS flipping file systems to "read only" right in the middle of something without any warning. Well then, you have not been properly introduced to a default feature in Linux whereby the kernel detecting an impending corruption on a disk will flip the file system to "read only" for protection. Initially my response was - "what in good God's name are we doing?", but then when I thought about this a little more I realized that this is actually a good thing IF we can eliminate false positives in the kernel actually detecting a real corruption on the disk as opposed to assuming one.
Let's look at this in a little detail. We all are aware that as far as matters of corruption are concerned it is better to be forewarned than suffer one. This is especially true if a corruption happens on a file system that houses large database files, the added misery of recovering the db is not fun. But no OS today actually does a real-time scan of disks to prevent this with 100% assurance. Though there are mechanisms in the DB and Storage to ensure block consistency most often these are not turned on due to performance impacts. So the way Linux initially used to check for corruption in real time is by measuring how long I/O operations take or by detecting storage path failures. So if the path fails to storage then it is pretty straight forward to detect but if you assume corruption based on some timeout of an I/O operation then you are in a grey territory. This is because how Linux defines latency for I/O operation may not cover you in case you have a really bad storage array which is extremely busy and has high service times. In such a case during periods of peak I/O like backup etc, Linux may see increased I/O response time and the SCSI timer in the kernel will timeout flipping the filesystem to read only even though there was no storage path failures or actual corruption. This is an irritation because when you go look you may not find anything wrong with paths or storage at the outset and you are left scratching your head. Once a filesystem flips to read only you can try remounting with read write flags, many times it may continue to be in read only state and the only method would be to reboot, run fsck (optional in case you don't see any real corruption) and check if it has mounted as read-write.
Now, some specifics, I have found that Red Hat Linux releases upto 4.7 have a higher sensitivity due to a bug 490744 , the issue is with the SCSI retries and timeouts especially if you have qla2xxx (Qlogic HBAs) drivers. The bug is fixed in release RHEL 4.8 (Linux release 2.6.9-89) and further improvements have been made in newer releases. Most times the file system involved in flipping to read only are ext3.
The key here is not to get mad at Linux for this feature, it is there to protect in case of real corruption issues. The fact that it flips on false positives is primarily because there is still an issue as far as the kernel and I/O latency or path failures. In most cases if you dig deeper you may be able to uncover suboptimal storage configurations, oversubscribed filers, misaligned LUNs (see my entry in this blog about misaligned LUNs under October 2012), misconfigured multipath settings, wrong MTU in case of NFS etc. that are causing very slow I/O performance. Once you deal with the underlying problems in storage you will see that things go back to normal and the number of flips to read only comes down.
Comments