Zbr's days.

About :: TODO :: Blog :: RSS :: Old blog :: Projects :: GIT :: Gallery :: Notes

Sat, 13 Sep 2008

New distributed storage release.

This is maintenance only release of the DST, which brings us following changes:

  • Fixed memory leak in crypto thread initialization error path. Noticed by Sven Wegener (sven.wegener_stealer.net).
  • Unprotected tree access (exceptionally stupid bug, I was made blind by the electronic equipment), and tricky bug_on catch in scsi code caused by incorrect bio flag initialization in the exporting node. 64bit alignment fix. Bugs reported by Rémy Ritchen(
  • Couple of bogus compilation warnings about unintialized variables cought by different compiler.
  • Allow both hread and write permission, not only read or write in security config.
Patch can be found in git tree or archive.

The most tricky bug is scsi's BUG_ON(), which did not even contain any DST related calls.
It was cought at drivers/scsi/scsi_lib.c:1175:
  kernel BUG at drivers/scsi/scsi_lib.c:1175!
  RIP: 0010:[]  [] scsi_setup_fs_cmnd+0x64/0x70
  ...
  [] ? sd_prep_fn+0xa8/0x9b0
  [] ? __cfq_slice_expired+0x59/0xb0
  [] ? cfq_dispatch_requests+0x8d/0x330
  [] ? elv_next_request+0x119/0x250
  [] ? scsi_request_fn+0x6b/0x3c0
  [] ? generic_unplug_device+0x24/0x30
  [] ? blk_unplug_work+0x41/0x80
Which is the following code:
int scsi_setup_fs_cmnd(struct scsi_device *sdev, struct request *req)
{
	struct scsi_cmnd *cmd;
	int ret = scsi_prep_state_check(sdev, req);

	if (ret != BLKPREP_OK)
		return ret;
	/*
	 * Filesystem requests must transfer data.
	 */
	BUG_ON(!req->nr_phys_segments);
Which means that request structure did not contain any segment to process. Origianlly I thought that it is because of some tricky elevator steps, which selected wrong request queue because of all debug showed, that sync bio (block IO request with BIO_RW_SYNC bit set) is handled differently compared to the same request without this flag. But experiments with various flags showed, that bug occurs no matter how, but just in completely unpredictible place.

Fortunately I managed to catch it in a debug trap in block IO merging path, which showed me, that block IO requests with very srtange read/write and flags fields was a cause of this error. Looking more precisely to the block queue allocation path, I found, that its default initialization is not correct, and my setup happens before it, so it did not contain the right parameters for the maximum request sizes (hw and phys sectors). This also showed, that one block IO request in the export node had clone and other local-only fields, which is very wrong for the bio to be submitted, which actually resulted in the seen bug. Those fields were set by the client bio and should not be transferred to the remote one, so I only limited flag fields to show that bio is uptodate and have blockable IO bit.

That's the story about how things were hacked this day (its a middle of the night actually, while I'm waiting for the taxi to move to the airport), so POHMELFS locking algorithm was not implemented today, and likely is postponed to the next weekend when I return, since I got a group theory book and made some prints about numbers theory (after completed reading Vinogradov's book), so I will have what to read in all four planes (two in each direction) if I will not fall asleep, and likely I will not have much time in Portland: we will need to talk/listen to other people and check local pubs (people suggested some coctail places, but I prefer beer).

See you in Portland.

/devel/dst :: Link / Comments ()