About :: TODO :: Blog :: RSS :: Old blog :: Projects :: GIT :: Gallery :: Notes

Thu, 18 Oct 2007

Added ability to store age of the node to the disk in mirror algorithm. New DST release.

This is used for the cases, when mirror node was updated (disks changed or something like that), so that media for failed node does not contain data, which was there previously. In this case dst core will read 'age' of the failed node (unique id stored at the end of the node, which is assigned to the whole storage during its initialization time), so if it does not match current one, the whole node will be marked as dirty and will force total resync.
The same applies to the initial setup - if id for the second or any subsequent node does not match id of the first one, nodes will be marked as dirty and will resync eventually.
This is a good step forward, I think. The only missing bit I'm thinking about right now is on-demand resync, i.e. when node found to be dirty. Right now resync only happens, when there are operations on top of the storage. This is quite minor priority though, as long as new redundancy algorithm.

/devel/dst :: Link / Comments (10)

Filesystem survey. Take 2.

I've started a short survey between my known admins about filesystem usage statistics they have on different systems, and about how it should be handled in the best case.
To make it clear I asked to specify type of the system (like mail server, e-commerce, database and so on), type of the load (a lot of small/big reads/writes, sequential/random access and so on), and expected results (i.e. what load should be preferred and what performance is expected and would be good to have). With results of the previous survey this should create a clear picture of the usage cases for any filesystem.

Feel free to drop in comments your ideas and experience with different systems.

Since those, who origianlly really wanted to participate in distributed system discussion, keep silence, I will postpone new redundancy code implementation (I started it in userspece already, so maybe I will return to it quite soon).
It also looks like vast majority of the systems of such scale use the simplest mirroring.

For the new release of the distributed storage I will definitely implement ability to store dirty/clean bitmap on the disk, mainly for the following purpose: suppose you have a distributed storage with multiple mirror devices, if one of them fails and you want to replace it with new one, which has new media, but higher layer (dst itself) only knows about pending updates, which were not stored on the media, so when new node 'resurrects' it will read its dirty/clean bitmap from the media, find that it is clear (or its age is not equal to age of the currently used system) and thus force DST core to copy all required blocks from different node.
This is not that complex task so I've started it right now.

Stay tuned.

/devel/fs :: Link / Comments (0)

New release of the userspace network stack.

Short changelog:

• really fixed leak in raw netchannel reading path
• changed timestamp setup
• added retransmit checking timer
• added sanity checks for addresses and ports processed in the stack - in case of packet socket they can be incorect some times (when working over loopback for example)
• retransmit logic checks - still requires bits of work, it is not 100% correct

/devel/networking :: Link / Comments (0)

Reading userspace network stack code.

	if (!th->ack) {
		ulog("%s: Strange packet.\n", __func__);
		goto out;
	}

/devel/networking :: Link / Comments (0)

We won! Russia - England 2:1.

Although I did not belive that - I checked radio at clibming zone after first half of the match - 0:1, so I lost the hope, but we managed to win.

Of course I'm very glad!

/other :: Link / Comments (0)