Zbr's days.
←July→
Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    
←2008→
Months
Jan Feb Mar
Apr May Jun
JulAug Sep
Oct Nov Dec

About TODO Blog RSS Old blog Projects Gallery Notes

Thu, 03 Jul 2008

POHMELFS crypto support has been completed. 

kernel$ git commit -a
Created commit b07e3ed: Added crypto support.
 9 files changed, 1534 insertions(+), 221 deletions(-)
 create mode 100644 fs/pohmelfs/crypto.c

fserver$ git commit -a -m "Aded crypto support."
Created commit f916b2f: Aded crypto support.
 3 files changed, 788 insertions(+), 94 deletions(-)
I implemented pool of crypto processing threads (number of them is mount option parameter), each of which has pool of pages to encrypt data into, so crypto thread is not released until server returns acknowledge that data was successfully written, so one should tune number of threads and page pool (number of pages in each thread is maximum number of pages per transaction, this limit has own mount option too) according to desired behaviour.

Testing shows that writing performance was reduced with this approach noticebly: with 4 encryption threads and 4 receiving thread in server perfromance dropped by around 30% from 65+ MB/s down to 46+ MB/s, but I think it can be improved with larger number of encryption threads. During iozone write/rewrite test each of 4 crypto threads ate about 20-30% of CPU, while server ate about 130% (4 threads totally). In all previous iozone tests the larger number of userspace was used, the worse results were (this is somewhat expected, since iozone is singlethreaded benchmark, so larger number of threads lead only to performance degradation), so I will test different setups (namely larger number of crypto threads and smaller number of server threads).

But this behaviour is not a problem, and I expect it to be tuned, real problem is reading performance. Right now there is only single thread, which reads from one socket: it was done intentionally, since reading data from socket is longer operation than searching page in radix tree or any other operation performed by that thread, so there is no way to saturate its capabilities. Until we start encryption, which is slow, so any subsequent data reading from the socket can not be done in parallel with crypto processing, and overall reading performance drops to ground.

This problem has to be fixed, so I plan to use the same crypto processing threads to decrypt and/or perform hash check for received data and push it up to the VFS stack.

/devel/fs :: Link / Comments (0)

Wed, 02 Jul 2008

POHMELFS crypto: feel incredibly stupid.

First, POHMELFS does need to have encryption. Because I plan to use distributed hash table approach in server (well, consider POHMELFS kernel client as a kind of bittorrent filesystem client), and as in any non-centralized system, content transferred via uncontrolled data channels has to be encrypted.

But... I'm incredibly stupid: I implemented encryption and decryption in place, i.e. VFS page is being encrypted prior to be written to the servers, so subsequent reading leads to... Yes, it reads encrypted content.
To fix this issue I plan to encrypt data into different pages and send them, leaving VFS ones as is. There are two approaches I consider:

  • allocate and send pages at writeback time - we want to send 5 pages, so allocate 5 pages, encrypt data into them and broadcast them to all needed servers.
  • allocate (potentially large) pool of pages at mount time per crypto thread and encrypt data into them. This will have about zero run-time overhead for VFS, except slightly delayed because of encryption write completion.

/devel/fs :: Link / Comments (7)

Mon, 30 Jun 2008

Filesystem development rumors.

Rumor number one. SWsoft aka Parallels actively searches for Linux kernel hackers in lead Moscow universities, namely MSU and MIPT. I saw theirs posters, where among other (wanted) requirements there is distributed filesystem knowledge.

Rumor number two. Alexey Kuznetsov (if you do not know, its the guy who wrote major part of linux network stack, namely TCP/UDP/IP and socket implementations, and although there was lots of changes in the stack since then, I think it will not be an exaggeration to call him the author), who also worked on Virtuozzo and OpenVZ (and its interesting VFS parts, which AFAICS are not in kernel, maybe yet), so he works on some filesystem too. The last time we 'confronted' was couple of years ago, when I first time implemented netchannels and tried to convince network community (and namely Alexey Kuznetsov and David Miller) that netchannel idea worth further investigation and implementation. IIRC I did not succeed, although results were very impressive.
Let's see what will happen with filesystems :)

Rumor number three. SWsoft recently started to actively search for kernel hacker for 'new interesting open source project'. They always searched for kernel programmers, but never told anything about projects, now something changed.

Rumor number four. OpenVZ and Virtuozzo have serious problems with NFS (especially when server dies), probably because of very ugly NFS protocol (yes it is), so its hard to properly virtualize it (or not?). There are no alternatives for NFS right now in major productions, but you all know about POHMELFS which right now can be used as really good replacement.

Rumor number five. SWsoft has long history of PHD defences (at least in MIPT) based on theoretical FS called TorFS (namely Tormasov FileSystem), year ago it was still not very alive project in practice, but I heard that it was very impressive in theory. This rumor exists really many years.

So, I have a quite clear picture, that SWsoft started development of the new distributed filesystem, which is aimed at first to replace NFS in virtualized environments. I can also imagine very interesting distributed parallel facilities needed for virtualized systems. And they try to attract lots of people to the project as long as really heavy artillery like Alexey Kuznetsov.

Which basically means, that sooner or later my development will meet strong concurency from this company, which has lots of really good professionals.
And that's very interesting and cool :)

P.S. or it may be a complete bullshit and delirium of my fevered consciousness.

And one fact about POHMELFS: today I finished client support for padded crypto processing of all requests and started to work out server bits, I expect to finish it in a day or around, so new release is very close.

/devel/fs :: Link / Comments (3)

Sat, 28 Jun 2008

Need to rethink POHMELFS crypto a bit.

1. Because of encryption problem - data to be encrypted has to be blocksize aligned, so some informaion about padding has to be added into network command as long as crypto data size.

2. IV generation. I decided to extend network command and put there 64 bit IV for given packet. using simple sequence number is enough to protect against repeat message attack.

3. Encryption/hashing data. I decided not to ecnrypt/hash network headers, and only do it for transmitted data. If transaction contains several commands, data for all commands will be encrypted/hashed, in case of hash, signle digest/hmac will be generated and placed into transaction header.

4. It is possible, that I will add strong header checksum, which will be generated only for header and placed into special field. It will be calculated assuming checksum field is zero. This step is optional so far, but network header has 32 reserved bits, which can be used for it.

Right now hashing and encryption work, but are not checked on server (although generated), because of crypto alignment ugliness I decided to rethink approach a bit.
Evolution process in action...

/devel/fs :: Link / Comments (0)

Thu, 26 Jun 2008

POHMELFS server got initial crypto processing capabilities.

POHMELFS server is able to handshake hash/cipher names and operation modes, to initialize appropriate algorithms and perfrom basic operations (like more generic hash_update() instead of different functions with different arguments used to hash data depending on operation mode, either simple digest or hmac: EVP_DigestUpdate()/HMAC_Update(). I'm working on the right way of doing crypto processing, since how it is done right now is a bit hairy, i.e. without serious changes in the code.
I already hate OpenSSL API: EVP_get_cipherbyname(), EVP_MD_CTX, EVP_DigestFinal_ex(). It looks like above functions were written by three different persons and they never actually talked to each other about how to make them look similar... But it is a minor issue of course.

So, when things are settled down, I will make a new release, likely it will see the light this week.

/devel/fs :: Link / Comments (0)

Wed, 25 Jun 2008

POHMELFS input crypto processing engine is ready for testing.

But testing can not be done without appropriate server support, which is now the main task. POHMELFS uses lazy crypto engine - each network state (it represents connection between client and one server) contains number of fields used exclusively for semi-lockless input data processing (it locks state when performs actual reading, but does not hold that lock when processing incoming messages, since it is the only path, which receives data), now it also has crypto information about how to manage reply messages (they include read page reply for example), so it does not queue work to be done by crypto threads, but does that itself instead. It may or may not be the bottleneck of the input path, tests will provide facts, so far I do not have plans to change it, but it can be done of course if performance will suck.

After I finish crypto processing in both client (it has been written, but requires lots of testing with server) and server (just have started to recall how to work with OpenSSL. Well, I've read how HMAC works in OpenSSL, found it to be simple enough and then started to read how to parse binary data in LISP :) But anything which is interesting for me now, ends up in good results for all other projects), I will switch to something different for a while.
Some voices in the brain ask to be spread it in lots of interesting directions :)

/devel/fs :: Link / Comments (0)

POHMELFS crypto performance.

I've ran read/reread and write/rewrite tests as described in previous run, now with HMAC(SHA1) of all outgoing transactions (note, that reading response data is not yet encrypted and does not contain digital signature, server also does not support neither operation), essentially only writing should be affected by this, but I also ran reading tests for compelteness.

Results show zero performance overhead of the full data SHA1 hashing, but note that quite fast machines were used (2 3Ghz Xeons (2 physical and 2 logical CPUs, HT enabled) with 1 GB of RAM). All the time only two crypto threads were actively hashing data, since there are only two pdflush threads on this machine.

Read Reread

Write Rewrite

Writing is even faster with hashing, but results drifted around, so essentially performance is the same.

/devel/fs :: Link / Comments (0)

Mon, 23 Jun 2008

POHMELFS client got initial part of multithreaded crypto/checksum processing.

So far it only includes encryption and hash calculation for outgoing transactions. System has (mount option) number of threads per superblock, which are responsible for encryption/hashing (each thread has own crypto structure, so there are no additional allocations in the fast path, although I think they would not harm performance since should be small enough fraction on top of crypto processing overhead) and subsequent data sending, so original caller (like writeback/readahead code) will not block if there are ready threads, otherwise it will wait until some thread finishes its current crypto work.

I decided to implement kind of continuation for such transactions, when network sending code (which is supposed to be started after crypto processing) will be invoked from those threads, which performed crypto operations, and not returning back to originall caller context. For massively multiqueue NICs that should be a benefit, but so far I did not test its performance.
Next step is receiving crypto support and userspace changes.

/devel/fs :: Link / Comments (0)

Crypto processing in POHMELFS. OpenSSL vs GNU TLS.

If I did not miss something, GNU TLS (I never worked with it) supports very limited amount of ciphers and hashes, so it is not appropriate for filesystem data protection layer.
According to its documentation GNU TLS only supports AES, RC4 and 3DES ciphers and SHA1 and MD5 hashes. There is also only CBC chaining mode and several hash/cipher schemes.

So, POHMELFS server will use OpenSSL for data protection. Sooner or later OpenSSL will get hardware crypto support on Linux too (well, Linux crypto stack should first implement userspace API, which does not exist yet, although there is a work by Loc Ho from AMCC to add such support).

So far I decided to implement following protection scheme: checksumm or encryption will cover full transaction data, but will be applied by chunks:

  • Transaction 'first-level' data, i.e. header and data immediately placed after transaction header. For all commands except page writing it will be finish.
  • For write pages command, each header is generated dynamically and does not exist until data is really being sent, so crypto code will run over all pages and update checksum processing headers and data pages separately. Checkum update should be simple enough, since there are crypto helpers to update and finalize checksum, but encryption is more complex: I requires all chunks to be setup in advance in single scatterlist chain, with dynamic header generation it is too big overhead (it requires not only scatterlist allocation, but also header allocation just for encryption), so encryption will be done separately for headers and pages, and I will have to create some IV propagation scheme (like last bytes of previous unencrypted chunk will become IV for the next chunk, or something like that). I understand, that it may be not very secure approach though.
  • Reading data back from server is simpler, since there are no transactions, and data will be encrypted/checksummed like in the first step above. It is possible, that it will force to increase network header structure a bit (32 or 16 bits to store size of the attached checksumm).

/devel/fs :: Link / Comments (2)

Thu, 19 Jun 2008

POHMELFS and HMAC/crypto operations.

As I found with distributed storage project, any communication channels, which involve huge amount of data transfers, have to have additional strong checksum embedded in the protocol, since TCP one is not enough in some cases. There are some options, like TCP MD5 signatures or IPsec transformations, but it is not always available.

POHMELFS will include ability to both encrypt whole data channel and/or only digitally sign all messages. This will be implemented on transaction level, so no higher layer code (like reading/writing data functions) will ever be affected.
POHMELFS will also have mount time self-configuration, i.e. client will send to server information about supported capabilities, requested by administrator, and if server does not support some of them (for example it can only do HMAC and not encryption, and both operations were requested at mount time), they will be dropped (and mount failed optionally). In the future it will be possible to extend it with additional flags if needed.

mount is not very convenient command to transfer crypto information (like binary keys) to kernel, so I use the same infrastructure as initial server group initialization (i.e. using POHMELFS existing configuration utility).

Support for HMAC and encryption will force server to depend on OpenSSL, but I do not think it is a problem. In some future time I can write autoconfiguration, which will allow to compile server without crypto support (and thus do not accept encrypted clients and do not check signatures) if there is no OpenSSL.

After crypto operations are implemented (I expect it to be finished this week), I will release as promised new netchannel version (and will remove unneded functionality like NAT), and add some interesting bits (like async processing) into distributed storage, so expect its new release soon too.

Stay tuned!

/devel/fs :: Link / Comments (2)

POHMELFS, NFS, Ext4 and XFS in iozone benchmark. Graphs.

Hardware used in testing: 4-way Intel E7520 system (two logical and two physical CPUs) 3Ghz 32 bit Xeons with 1gb of ram, Adaptec AIC7902 Ultra320 SCSI adapter with SEAGATE ST3300007LC 10k rpm 300 Gb testing disk. Its linear reading speed is about 90 MB/s.

Software used in testing: 2.6.25 kernels (on server and client), in-kernel async NFS server, userspace POHMELFS server.

Tests were performed with 8gb files (amount of ram was reduced to 1gb to eliminate caching influence) with different (from 8 to 1024 KB) record size. I ran write/rewrite, read/reread and random read and write tests.

Read Reread

Write Rewrite

Random read Random write

/devel/fs :: Link / Comments (0)

CRFS got metadata cache coherency support.

Zach Brown has committed cache coherency support into CRFS repository.
Cache coherency protocol works by broadcasting special messages from server, and each client invalidates appropriate inodes (and dentries if needed) before sending back a reply.
POHMELFS uses a bit different mechanism: client does not send acks back to server, so all such messages are kind of advisory-only, but I did not yet complete (well, I did not even think about this problem this week) locking design, so it can change.

Main problem with sync cache coherency support is its absolute non-scalability. While number of sage cases might require such behaviour, I expect that if not major, but noticeble part of users do not want perfromance degradation as a price for posix-like coherency expectation. This approach is worse that write-through cache, since there is whole round-trip of the cache coherency request instead of just data sending during its writing. Single direction sending is faster than sending+waiting, so for me it is still a questionable approach.

I will think a lot of this problem later this week(end), so that solution would satisfy both high-perfomance and safety camps (although at some degree only I think).

/devel/fs :: Link / Comments (0)

Fri, 13 Jun 2008

The latest iozone benchmark of POHMELFS, NFS, XFS and Ext4.

1Gb of RAM, 8Gb files. SEAGATE ST3300007LC 10k rpm 300 Gb on Adaptec AIC7902 Ultra320 SCSI adapter.

Performance in KB/s.

NFS:

                                                   random  random
     KB  reclen   write rewrite    read    reread    read   write
8388608       8   53210   57769    24304    24448    1360    4775
8388608      16   54577   57481    23871    24080    2592    7937
8388608      32   54736   56203    24015    24114    4738   12637
8388608      64   52075   54051    23653    23555    7610   18475
8388608     128   52307   54636    23305    23375   13017   26584
8388608     256   52189   53030    23585    23531   15615   34390
8388608     512   52938   54063    23709    23882   17524   42781
8388608    1024   57458   57006    24187    24292   29701   43892
POHMELFS:
                                                   random  random
     KB  reclen   write rewrite    read    reread    read   write
8388608       8   66473   63721    74232    74288    1103    4953
8388608      16   52604   62339    73423    74259    2001    8438
8388608      32   53278   62283    73497    74115    3360   13849
8388608      64   56931   61370    73135    74077    5076   21063
8388608     128   59419   62743    72736    74122    8068   30279
8388608     256   60861   63094    73284    74554   10848   38869
8388608     512   59438   62081    73329    74441   17290   48722
8388608    1024   62790   62130    73322    74100   27741   46470
POHMELFS write speed about 10% faster, read speed 3-3.5 times faster (essentially disk/local fs IO limit, see below). POHMELFS random read speed is smaller, and that is task with the highest priority now, especially compared to local FS results.POHMELFS random write is slightly faster than NFS.

For comparison, local filesystem, used for tests.
mkfs.xfs -d agcount=75 -l size=64m /dev/sdc1;
mount -o logbufs=8,nobarrier,noatime,nodiratime,osyncisdsync /dev/sdc1 /mnt/:
                                                   random  random
     KB  reclen   write rewrite    read    reread    read   write
8388608       8   75124   60560    77672    77797    1860    5059
8388608      16   75044   60036    77754    77775    3601    8772
8388608      32   75958   62038    77593    77765    6821   14781
8388608      64   74728   59384    77688    77782   12475   23228
8388608     128   74889   59676    77731    77736   21734   32241
8388608     256   75022   59285    77676    77718   28833   40324
8388608     512   74885   59187    77653    77713   40013   48057
8388608    1024   74838   64217    77796    77765   55100   46104
And Ext4 to the group (mount options: rw,noatime,data=writeback,extents):
                                                   random  random
     KB  reclen   write rewrite    read    reread    read   write
8388608       8   72107   73017    77276    77335    1849    5015
8388608      16   72276   73849    77304    77287    3577    8666
8388608      32   72680   73647    77284    77326    6755   14394
8388608      64   71965   74287    77327    77288   12366   22513
8388608     128   72660   73864    77207    77343   21617   31160
8388608     256   72813   74058    77296    77338   28652   42003
8388608     512   72985   73317    77284    77343   40572   50619
8388608    1024   72184   74131    77264    77250   55649   50365
Nice graphs will be done, when I will write Lisp (no less :) parser for it.
Stay tuned!

/devel/fs :: Link / Comments (3)

New POHMELFS release: doing it wrong fast is at least better than doing it wrong slowly.

Via Ashleigh Brilliant and bits of Tullamore Dew.

Here we go, short changelog for this release:

  • Read requests (data read, directory listing, lookup requests) balancing between multiple servers.
  • Write requests are sent to multiple servers and completed only when all of them sent an ack.
  • Ability to add and/or remove servers from working set at run-time from userspace (via netlink, so the same command can be processed from real network though, but since server does not support it yet, I dropped network part).
  • Documentation (overall view and protocol commands)!
  • Rename command (oops, forgot it in previous releases :)
  • Several new mount options to control client behaviour instead of hardcoded numbers.
  • Bug fixes.
I will complete documentation in a few moments and send this release to the mail lists.
Very likely it is last non-bug-fixing release of the kernel client side, next release will incorporate features, needed for distributed parallel data processing (like ability to add new servers via network command from another servers), so most of the work will be devoted to server code.

/devel/fs :: Link / Comments (0)

Wed, 11 Jun 2008

Preparing for the next (last non-bug-fixing?) release.

Essnetially that's it, I belive really most of the features I wanted from network distributed parallel filesystem, which should live in client, are already implemented in POHMELFS.

Client has following (if did not forget something interesting, listed only interesting from parallel point of view) features:

  • Automatic failover reconnect to the same server.
  • Run-time addition/removal of the servers from the working set (only via userspace command, since server does not support that yet, but addition is trivial).
  • Coherent data and metadata cache
  • Transactions support. Full failover for all operations. Resending transactions to different servers on timeout or error.
  • Load balancing of reading (directory reading and lookups inclusive) requests and simultaneous writing to all servers in current working set.
It is damn fast (but remember, that random reading is no yet optimal enough, and in the last tests it was slower NFS).

Userspace server meantime does not support lots of features it has to support to be called complete parallel distributed solution, and main work should now be concentrated on it.
Main missing (and the most complex) features are:
  • Distributed data coherency protocol like PAXOS for server data, stored on multiple machines.
  • Ability to mirror data itself on multiple machines.
So, likely release will see the light tomorrow or Friday.

/devel/fs :: Link / Comments (0)

Fri, 06 Jun 2008

POHMELFS development status.

POHMELFS got ability to add/remove servers in run-time (although not via network command, since I do not know, how to test it yet), but via netlink interface. The same message can be passed via network though, so it will be simple to extend.
Also, POHMELFS got readahead support via ->readpages() callback. I removed AIO reading from POHMELFS in favour of readahead and got excellent result in sequential reading: 3-3.5 times faster than NFS and essentially reaching disk IO bandwidth (a bit less though), but random reading dropped to miserable numbers.
Also rewritten reading method should provide better balanced between multiple servers capabilities for the system, but it will not show any benefit in single-threaded iozone benchmark, since it reads data via single call to read(), which gets sequential data access, which in turn is faster than network bandwidth. So multithreaded load should greatly benefit from read balancing, but I did not yet test that.

I ran sequential read/reread, write/rewrite and random read/write tests for XFS, Ext4, NFS (over XFS) and POHMELFS (over XFS) with 1Gb of RAM and 8Gb of test files (to eliminate VFS caching influence) with 8Kb to 1Mb record size.
Results exist in text files in standard iozone output format, but since I'm learning LISP I decided to write a graph generator (via gnuplot) using my very basic knowledge of this language, so nice graph results can take a while...

Also, tomorrow morning I will flight away to my friends marriage and will only return monday 9. I will not have internet access there, only lots of fun.

/devel/fs :: Link / Comments (0)

Wed, 04 Jun 2008

Optimized POHMELFS transactions.

Now they eat less memory, and single writing transaction can accumulate up to 1024 pages. This can be further tuned especially for small requests mixed with sync. Currently writing transaction is allocated for its maximum size, and then pages pointers are written to the allocated area, so if number of dirty pages requiring writeback is small, quite lots of space will be wasted.
It is a task for the next optimization, nevertheless currently sequential writing is only limited by disk throughput or network bandwidth in case of multiple servers, since link is shared between machines, so effective bandwidth becomes equal to GigE/number of servers, or about 60 MB/s in my environment with two servers and single client.

Also, reading path was not changed at all (only transaction internals) - there is still no readahead and new transaction is allocated for each page to be read. Nevertheless, see how reading was improved: POHMELFS not only outperformed NFS again, but reached disk bandwidth limit already for 16Kb requsts (almost two times faster than NFS). Table shows IO throughput in KB/s.

                                                    random  random
      KB  reclen   write rewrite    read    reread    read   write
 8388608       8   74058   68392    40130    79509   43588    4818
 8388608      16   62332   66978    73714   122074   42160    8434
 8388608      32   64775   67073   109357   171139  145416   14183
 8388608      64   66962   66602   147350   217323  227962   22257
 8388608     128   67724   67133   185574   266855  321060   32681
 8388608     256   68233   67922   201591   283567  474657   40944
 8388608     512   68339   66514   213513   295995  646897   50303
 8388608    1024   67744   67384   220858   297748  676582   48796
I will create nice graphs out of this tables and also will include optimized reading tests (tomorrow likely) and two data server results.

What also should be done, is testing with either bigger files or smaller amount of ram and thus smaller VFS cache size. As you saw in all tests, when lots of reads start to hit the cache, picture becomes completely non-informative for filesystem behaviour. So I want to limit all three testing machines to 1Gb of RAM (booting with mem=1G parameter) and perform the same iozone bench for 8Gb file. Results should be more realistic.

In parallel I will implement userspace run-time server addition/removal command, which will also be used as-is for network message from one or another server, connected before. With optimized reading transactions it will be a good ground for the next POHMELFS release. So I plan to schedule it to thursday or middle of the next week, since I will be on small vacation jun 6-9.

/devel/fs :: Link / Comments (0)

Mon, 02 Jun 2008

AppArmor and path-based security approaches vs object bound policies.

- So again, can you offer an alternative?
- Just give up on this dumb idea completely.
It is not about AppArmor in general (although maybe about it too), but about security hooks which provide path information into inode callbacks. There are pros and cons for this decision, but things look like path based security hooks will not be accepted.

There is a really trivial way to fix it. No kidding, it is simple: create own name cache and do not bind it to dentries, but instead index it by inode number. This allows you to have whatever you want callbacks and information in stricktly bound VFS operations. Need to have path info in ->inode_create()? Put it into own tree indexed by inode number for parent inode, lookup that data in security hook and make a decision. Yes, it is slower, but active security was never a fast solution. It is still against the rules others created for security based systems, but still formally it in the all boundaries of the created (maybe ugly for someone) interfaces.

And I will not point to project, which already uses such approach in different area though :)
It is interesting to implement your ideas not by breaking something (although sometimes it is need, but that's likely an exeption or when you are hacking deeply internal kernel part), but instead by hacking around existing limitations.

/devel/fs :: Link / Comments (4)

As promised, let's see shadowed miserable POHMELFS results.

Usually you will not see bad benchmark results for developing technology, but any such result is actually a _very_ good result for work-in-progress and not yet completed system. It allows to see how new proof-of-concept code can be comparable with already completed tuned and optimized system.
Conclusions from such test results in a really superior decisions.

Let's compare iozone read/reread, write/rewrite and random read and write for POHMELFS and NFS with 8Gb test files different record size (from 8Kb to 1Mb) on XFS over the GigE link.
I described hardware and local iozone benchmark results in details previously.

Now its time for network tests.
Async NFS in-kernel server results.

						    random  random
      KB  reclen   write rewrite    read    reread    read   write
 8388608       8   60969   57743    39705    97031  464898    5160
 8388608      16   59925   57402    39045    98269  641388    8827
 8388608      32   58094   55263    39075    94654  775064   14389
 8388608      64   58168   57156    40306    98639  868796   22360
 8388608     128   58908   56573    40392   100018  941509   33211
 8388608     256   59444   56446    40842   102503 1030451   41576
 8388608     512   60280   57686    39835    97879 1042570   49858
 8388608    1024   60817   57886    40886    96646  851175   47993
And now POHMELFS results.
						    random  random
      KB  reclen   write rewrite    read    reread    read   write
 8388608       8   70073   64232    12518    14817   40334    5079
 8388608      16   63984   67948    31976    19106   41462    8702
 8388608      32   67250   63440    47506    38657   75908   14357
 8388608      64   69970   66198    41899    29566  136294   21385
 8388608     128   69838   68523    76232    33971  222909   30946
 8388608     256   70012   66439    69125    58223  330886   40685
 8388608     512   70946   68291    76460    58738  428881   51001
 8388608    1024   70985   64958    76317    59561  421973   48531
Sequential writing is 10-15% faster for POHMELFS (and limited by underlying fs speed), while random writing is essentially the same and is limited by disk speed. But sequential reading is _much_ worse for small requests. THe reason is simple: POHMELFS does not support readahead, since it does not have ->readpages() callback, so any sequential access ends up with set of ->readpage() callbacks, which waits for theirs completion, which is slow, so currently readahead is not invoked from reading path.
I could not resist to highlight, that big sized requests are 1.5-2 times faster for POHMELFS than NFS :) and is also limited by underlying filesystem.

One can note, that NFS random reading results are actually better than local filesystem behaviour, and its is better very noticebly. Why does local filesystem behave worse than being mounted via NFS in random reading?
I believe that's because in a network case we actually have double buffering: on client, where the most active pages are in RAM, and on server, where readahead populated pages, which are not active (since active pages are being read from client's cache, so they will be evicted from server's page cache, since client will not try to read them from server), but those server pages, which are not active currently will be accessed soon by client, when it will read next portion of the random data, and it will be very fast access to RAM.
So we have really good caching scheme, where the most actively used pages are in client RAM, and they are flushed to disk on server, and isntead server populated other less active pages via readahead.

This reading behaviour is just a result of yet not completed VFS callback implementation of the POHMELFS. With ->readpages() in place it will be faster than NFS even in this bench. Also POHMELFS has multiple-server parallel read balancing and simultaneous writing to them, but there are no results yet.
I already created a mind model of the optimized read and write transactions (based on memory pools for the maximum OOM-robustness and small memory usage overhead), so in a day or two it will be implemented in code.

Stay tuned, now its time for excellent POHMELFS results!

/devel/fs :: Link / Comments (0)

Fri, 30 May 2008

Local filesystem randomg read/write performance. POHMELFS parallel testing.

I promised to publish POHMELFS parallel processing results yesterday, even if they are miserable. Unfortunately there are no interesting results at all. In the released version POHMELFS is 32bit only, since it does not have special ->open() callback which forces to open files with O_LARGEFILE flag to support more than 4Gb (actually only 2Gb, since kernel uses signed size_t, which is only 31 bit large) sizes and superblock maximum size is set to 32 bits, so all 32 bit results are not very interesting, since having 2Gb/s random read speed is really stupid sentence, since all reading happend from the cache.

While results with more than 2Gb are... Let me first show you how XFS and Ext3 behave in case of random writes.

A short preface.
Hardware used in testing: 4-way Intel E7520 system (two logical and two physical CPUs) 3Ghz 32 bit Xeons with 8gb of ram, Adaptec AIC7902 Ultra320 SCSI adapter with SEAGATE ST3300007LC 10k rpm 300 Gb testing disk.
Its linear reading speed is about 90 MB/s. Dmesg:

scsi0 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 3.0
        <Adaptec AIC7902 Ultra320 SCSI adapter>
        aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
scsi 0:0:2:0: Direct-Access     SEAGATE  ST3300007LC      0003 PQ: 0 ANSI: 3
 target0:0:2: asynchronous
scsi0:A:2:0: Tagged Queuing enabled.  Depth 32
 target0:0:2: Beginning Domain Validation
 target0:0:2: wide asynchronous
 target0:0:2: FAST-160 WIDE SCSI 320.0 MB/s DT IU QAS RDSTRM RTI WRFLOW PCOMP (6.25 ns, offset 63)
 target0:0:2: Ending Domain Validation
Kernel version is 2.6.25 (and 2.6.24 for the first ext3 test).

I used two such machines as servers for iozone read/reread, write/rewrite and random read/write testing. File size is limited to 8Gb only, since it is the only interesting fair case, record size varies from 8Kb to 1Mb.

Before I started 8Gb POHMELFS testing, I decided to check how local filesystem behave in such scenario.
XFS was tuned this way: (mkfs.xfs -d agcount=75 -l size=64m /dev/sdc1; mount -o logbufs=8,nobarrier,noatime,nodiratime,osyncisdsync /dev/sdc1 /mnt/)
Ext3 was created and mounted with default options on machine with only 4Gb of RAM though.

So, testing.
Here is a results table from iozone (before I interrupted it) with read/reread, write/rewrite and random read/write tests for XFS (either default, or tuned like on link above).
                                                   random  random
     KB  reclen   write rewrite    read    reread    read   write
8388608       8   73671   64052    77565    80107   35281    5085
8388608      16   74437   66095    77611    80065   66854    8808
8388608      32   74683   66780    77564    80202  121442   14576
8388608      64   74936   66908    77537    80372  215377   22583
8388608     128   74928   68598    77542    80247  339304   32280
8388608     256   73609   69615    77534    80143  365081   40571
8388608     512   73763   69830    77547    80317  420704   48501
8388608    1024   73940   69474    77602    80065  406266   47295
I.e. 5 MB/s random write speed for 8kb record!

Do you really want to know ext3 speed? Pregnant kids and women should skip next paragraph.
I interrupted test after almost 2 (!) hours or random writing of 8Gb file with 8Kb records on default ext3. Test was not completed and I do not really know its performance (note, that this machine has only 4Gb of ram, other hardware details were described above), but it will be less than 1 MB/s.
Ext4 behaves much better in this aspect (ount options: rw,noatime,data=writeback,extents):
                                                   random  random
     KB  reclen   write rewrite    read    reread    read   write
8388608       8   69593   74200    77324    81340   35538    5088
8388608      16   66745   70038    73676    77271   65715    8704
8388608      32   68253   70320    73652    77258  121690   14469
8388608      64   68421   71291    73653    77042  209629   22005
8388608     128   68438   71340    73658    76988  332021   30381
8388608     256   68921   71254    73651    76912  435586   40683
8388608     512   69079   71728    73551    76815  549136   49298
8388608    1024   66611   71217    73683    76581  552459   49220
POHMELFS results are coming...

/devel/fs :: Link / Comments (0)

Wed, 28 May 2008

POHMELFS got read balancing between multiple server and simultaneous write to them.

I hate laziness, but sometimes drop into that hole... So last couple of days I just stupidly wasted by time (well, I read Lisp and failed to find GTK binding for CLISP, made some code and kernel bug fix, but that does not count). Today lazyness started to be really boring, so I made some small progress in POHMELFS parallel processing.

It got ability to send transactions to multiple servers by default and balance reading between them (so far it does it always from the first server, in case of error it switches to second, but it is trivial to change). This was implemented via special routes for each transaction, which are stored per network state, so if one of the servers did not answer, we would not resend data to others. It also makes trees smaller, which should allow faster reading in case of lots pending writing transactions.
Code is in testing stage currently, I will complete read balancing tomorrow and test it against multiple servers on different machines, when data is placed on disk, so that random access would be slow. Having two servers I exect to get linear speed increase. If test will be disk IO bound, it is possible to add multiple servers on the same machine, so that each server would run on its own disk (I have two resonable fast SCSI disks on each testing machine).
Results will be published here of course (well, even if they are miserable :).

/devel/fs :: Link / Comments (0)

Sun, 25 May 2008

New POHMELFS release. Full transaction support. Data and metadata cache coherency.

Irish Tullamore Dew helped this POHMELFS release to see the light.

Short changelog:

  • Full transaction support for all operations (object creation/removal, data reading and writing). Data reading transactions are not optimal yet and will be improved in the next release (although fast).
  • Data and metadata cache coherency support. More details on how this is implemented one can find in appropriate section.
  • Transaction timeout based resending. If given transaction did not receive reply after specified timeout, transaction will be resent (possibly to different server).
  • Switched writepage path to ->sendpage() which improved performance and robustness of the writing.
  • Preliminary support for parallel data processing. Code to write data to multiple servers in parallel and balance reading between them was imported, but is not used right now.
  • Fair number of bugfixes.
Next release is scheduled for the beginning of the next month, and will likely include following features:
  • Improved reading transactions.
  • Server redundancy extensions (ability to store data in multiple locations according to regexp rules, like '*.txt' in /root1 and '*.jpg' in /root1 and /root2.
  • Client parallel extensions: ability to write to multiple servers and balance reading between them. Code was imported to the current version, but not enabled yet.
  • Client dynamical server reconfiguration: ability to add/remove servers from working set by server command and from userspace.
  • Start generic server distribution development.
As usual one can grab the latest source from archive or GIT tree.

/devel/fs :: Link / Comments (0)

Sat, 24 May 2008

This was supposed to be POHMELFS release day.

But no, it is scheduled for tomorrow because of the very interesting way I decided to implement reading transactions. The way it works right now is quite miserable, so I want to clean things up and make a really good patch.

Page reading code will create single transaction for the bunch of pages and will schedule next one if pages are not yet received instead of waiting for transaction to be completed, and only wait at the very end (if needed). With addition of async copy from receiving kernel thread into reading userspace via copy_to_user() (in todo), this will became the fastest possible way of doing reading over the net I think.

So far changelog contains following items:

  • Full transaction support for all operations (object creation/removal, data reading and writing). Data reading transactions are not optimal yet and will be improved in the next release.
  • Data and metadata cache coherency support. More details on how this is implemented one can find in devel section.
  • Transaction timeout based resending. If given transaction did not receive reply after specified timeout, transaction will be resent (possibly to different server).
  • Switched writepage path to ->sendpage() which improved performance and robustness of the writing.
  • Fair number of bugfixes.

/devel/fs :: Link / Comments (0)

Wed, 21 May 2008

iput() locking in POHMELFS.

iput() is a very tricky call in Linux VFS, besides the fact that it drops inode when its reference counter reached zero, it also waits until all associated pages are flushed to storage too.
POHMELFS uses singler per network state (network connection structure) thread, which only reads async replies from the server, so it is possible, that reply which requres iput() (for example create command reply) will happend in parallel with object removal, so inode will be deleted, but yet not freed. When reply is received and iput() called, it will try to free inode and wait until all associated to its mapping pages are synced. But page sync happens on reply to another command (consider for example several writeback transactions), which can not be processed, since thread is waiting them to be completed. This problem can not be fixed by introducing multiple threads, since each one can be exactly in the same situation simultaneously.

In turn we should not allow to grab inode and free it in the receiving path. This is ok for writeback transactions, since inode can not be freed until pages are synced, so just by holding pages we are able not to lock, but object creation for empty files or directories does not have pages attached, so they have to be synced with special transaction. There still can be a problem with empty file though - some pages can be attached and it can be removed while system waits for creation transaction complete, but actually we do not need to know about that - we shuold not grab inode it all, since transaction already contains all needed into, namely inode number, so we can lookup inode (if it still exist) and mark it as created without need for lock-prone grab/put.

This bit took me last three days, during which POHMELFS moved to non-blocking receiving and timeout-based sending (and returned back), it got scanning 'watchdog' which resends trasactions if they were not acked after some time and eventually dropes them if they still does not get a reply, POHMELFS got couple of new operations supported and likely something else to existing set of features implemented to date (full transaction support for all operations and data and metadata coherency protool were added for the next release).
New release is scheduled for the end of the week, and there is no readpage transaction support yet...
So, stay tuned!

/devel/fs :: Link / Comments (3)

Sat, 17 May 2008

POHMELFS got full data and metadata cache coherency support. Transaction support for majority of the commands. 

linux-2.6.pohmelfs$ git-diff-tree -r --stat 21549d0a101 master
 fs/pohmelfs/dir.c   |  108 ++++++--------------
 fs/pohmelfs/inode.c |  279 ++++++++++++++++++++++++++++++++++++--------------
 fs/pohmelfs/net.c   |  216 ++++++++++++++++++++++++++++++---------
 fs/pohmelfs/netfs.h |   43 +++++++-
 fs/pohmelfs/trans.c |   55 +++++++++-
 5 files changed, 484 insertions(+), 217 deletions(-)
It was rather simple task due to async event processing support.
Each time client creates, reads or writes object to server, information about its interest is stored on server. When any other client updates the same object (like changing attributes or writes data), all interested clients get notifications with new data (new attributes, or in case of writing possibly new size and flag, which page has to be fetched from the server, since it is not valid anymore). Writing happens during writeback as before, so commands like "echo Some_message > /mnt/file" immediately syncs size of the file to zero and after some time writes there actual data, when system will decide to start writeback.

Also ported all but one commands to transaction mechanism, which means they all will be resent if currently active network connection goes down. Although most of the commands are not synchronous, and thus will not be resent after timeout, this can be trivially changed if there will be major demand on that.

Only reading has not yet been ported to transaction model, which is a next task to complete. This transactions have to be synchronous, since we do want to read data, while do not actually care about full directory content.

This changes have to be seriously tested and all problematic places to be resolved, for example they slow metadata operations noticebly, since now system sends a message each time new object is created, although kernel archive untarring now takes about 5 seconds against previous 2-3 including sync on 4-way machine with 8gb of RAM and it is still not comparable to 30+ seconds for async NFS, it has to be investigated further.

After full move to transaction model and cache coherency testing (that model may be not complete for some usage, since locks are not yet supported), POHMELFS will make its first steps into distributed area...

Stay tuned!

/devel/fs :: Link / Comments (0)

Fri, 16 May 2008

Metadata cache coherency support in POHMELFS.

Client:

$ ls -lai /mnt/test
3 -rw-r--r--  1 root root 94208 2008-05-16 22:27 test
$ sudo chown zbr.zbr /mnt/test 
$ ls -lain /mnt/test
3 -rw-r--r-- 1 2319 1002 94208 2008-05-16 22:27 /mnt/test
Server:
fserver_get_client_data: thread: 3085847440, cmd: 8, id: 0, start: 2, size: 94, ext: 0.
fserver_transaction: thread: 3085847440, trans: 0, size: 94, sub: cmd: 10, id: 3, start: 0, size: 70, ext: 6.
fserver_inode_info: path: '/test', size: 94208, mode: 100644, uid: 2319, gid: 1002.
So, server now contains all metadata information about updated object on client, pohmelfs_setattr() is synchronous for remotely read inodes and for already synced indoes, created originally locally. It does nothing, if object is not yet synced to server, since syncing will provide that info itself.

The only missing thing is to asynchronously broadcast that data to other clients, which requires to create a cache of objects to be interesting for given client, each client will be automatically added into group of interests when it lookups object, so when attribute for given object is being set, update will be sent to interested parties. Client will be dropped from group of interests, when it drops appropriate inode locally (which will force sending a special message).

/devel/fs :: Link / Comments (0)

Thu, 15 May 2008

POHMELFS distributed plans.

After healthy discussion started after my announcement of the second POHMELFS release, its time to highlight main ideas settled in the thread.

First, POHMELFS will be moved into parallel distributed filesystems, but still being very good as network filesystem. In particular, that will include ability to read data from one of the connected server (not particulary from currently active, how its done right now), writing will happen to all connected servers simultaneously (and transaction will be committed after all servers returned completion acknowledge).

Protocol will be extended to support dynamic addtion and removal of the servers to/from currently connected group. Probably there will be some kind of a status messages for servers (i.e. going offline, do not send me data, or I'm becoming slow, do not read from me and so on). It will be done in addition to cache coherency messages (I'm yet to implement, but because of other tasks, this was a bit postponed, probably to weekend), which will include two types of requests: page invalidation and inode update (that will also mean that POHMELFS will start supporting attributes (maybe even extended), right now it doesn't :). Such cache coherency protocol should scale better than classical MOSI (and its derivatives) and particulary better than pNFS spec proides (leases to operations for some servers), since it is still possible to work in parallel with the same file, especially without any overhead of data processing does not cross different client boundaries, but it has to be tested in practice.

POHMELFS server will be extended to support distributed facilities. Very likely it will be some kind of PAXOS algorithm, although probably in its very limited mode for the beginning. So far it will be really simple, so that I could touch all its corner cases and found optimal development strategy.

All client extensions are rather not that complex, although not always trivial, so that should not take too much time, so probably you will get something interesting soon.
Server extensions will be a bit slower, since I will start essentially from the distributed system ground and gradually move upstairs.

/devel/fs :: Link / Comments (0)

Tue, 13 May 2008

New POHMELFS release. Transactions, performance, failover.

Irish Jon Jameson (6 years of experience, really good stuff) brings us this new POHMELFS release.

Main features include:

  • Fast transactions. System will wrap all writings into transactions, which will be resent to different (or the same) server in case of failure.
  • Failover. It is now possible to provide number of servers to be used in round-robin fasion when one of them dies. System will automatically reconnect to others and send transactions to them.
  • Performance. Super fast (close to wire limit) metadata operations over the network. By courtesy of writeback cache and transactions the whole kernel archive can be untarred by 2-3 seconds (including sync) over GigE link (wire limit! Not comparable to NFS).
The nearest roadmap includes:
  • Full transaction support for all operations (only writeback is guarded by transactions currently, default network state just reconnects to the same server).
  • Data and metadata coherency extensions (in addition to existing commented object creation/removal messages).
  • Server redundancy.
One can check out POHMELFS homepage for more details. You can download latest release (against 2.6.25 kernel tree) from archive or GIT tree.

/devel/fs :: Link / Comments (0)

Mon, 12 May 2008

Fast POHMELFS transactions.

With new transactions and new waiting mechanism (see below) system now untars the whole kernel tree in less than 3 seconds over the GigE link (including subsequent sync, which takes less than second always), while async NFS (remote side is tmpfs in both cases) performs that in a bit more than 30 seconds. In addition POHMELFS write speed is 125 MB/s (wire limit) vs. less than 90 MB/s in NFS (dd from /dev/zero with 1 MB block size and 1000 blocks).

That's what I call a good result.

Transaction mechanism invoked in writeback path is now completely async too, i.e. it does not wait until remote side confirms that transaction was received and processed, but writeback does not drop transactions after sending function returned, instead it stores it in the in-flight storage and proceeds with the next one. Transaction can accumulate up to 90 pages in a single frame.
When reply is received, async thread searches for given transaction and complete it (unlocks page, although it can be done in writeback, since page is being copied, cleanup writeback bits, drops it from appropriate radix tree and drops reference counter). If transaction was not sent due to some error it will be tried to be sent to different servers, if some error was returned from the server, it will be resent to different ones. Since original writeback path does not know about transactions in-flight anymore, any timeout has to be checked by dedicated thread (or workqueue), which will detect too old transactions (by simply checking them from the beginning, since each new transaction has incrased id) and resend them to remote servers.

There is a small problem though - if object size is more than single transaction can accumulate (90 pages), it will be split into several transactions, where first one will contain object creation command and some data to be written, while others will contain only data. If server runs multiple threads per client (default is one though), it is possible that not first transaction will be processed first, so server will write some data into non-existent file, so transaction will fail. There are two ways to fix this isuue: either wait in writeback on client while creation transaction is completed, and then send all others like described above, or add creation command into every subsequent transactions until object is created on the server (special bit is set on local inode in that case). Likely the latter is better case.

/devel/fs :: Link / Comments (0)

Wed, 07 May 2008

Fast transactions in POHMELFS.

POHMELFS just switched to faster transactions allocated one-by-one with even smaller overhead (although it does not use kernel_sendpage() for page sending yet, it copies data).
System does not serialize after all transactions are completed (it waits after each one), but with new transaction allocation it is 1.5 times faster: 98MB/s vs. 64MB/s, note that without waiting for transaction completion it gets full wire speed of 125MB/s with 1500 byte MTU. And it is with highmem pages and thus slow kmap() of each one, and unmap after completion. I do not use ->sendpage() since it will force to split proper set of iovecs into mixed calls of kernel_sendmsg() and kernel_sendpage(), which I want to avoid so far. Now it is (again) faster than NFS, but I want to move further.
So, solution is rather trivial: wait until several transactions are completed. There is the whole infrastructure already there - in-flight transaction storage, per-transaction completion and destruction callbacks, proper reference counting and async completion.
Still only writing transactions are used (i.e. reading/lookup and others will not redirected to different servers).
There are some bugs of course, but that's the first development version after all.

/devel/fs :: Link / Comments (0)

Mon, 05 May 2008

POHMELFS transaction support. Failover (re)connection to different servers.

POHMELFS just got full transaction support. So far it is only used in ->wrteipages() callback, which is invoked by writeback mechanism. POHMELFS uses lazy transaction support, namely it waits after each transaction, which includes header and data to be written for at most 14 pages, 14 is a magic number of pages, which corresponds to struct pagevec size, used by generic writeback, transaction size is limited by mount option and is 32 pages by default. Performance was dropped from 125 MB/s down to 64 MB/s, which is not acceptible. Main problem is of course waiting for transaction to be completed (i.e. completion message from server). There should not be per transaction waiting, instead writeback has to allocate as much transactions as needed and proceed one after another, and only start waiting for them, when there are no more pages to be written. This is the next task.

Transaction mechanism allows quite simple reconnection to different master servers in case of failure, and rollback of the failed transaction. For example one can provide different number of main servers (which have to be in sync with each other and be able to be synchronized themselfs, or they just can use shared storage), so POHMELFS client will switch between them if current one has failed. System will detect it and reconnect, if reconnect fails, next server will be used and the whole transaction will be resent there.
It is also possible to write transaction to different server on demand (it may or may not to be connected already, but it has to have address structure, so far it is only obtained during pre-mount configuration), which is a prerequistic for parallel data processing. One can create a simple patch to write transactions one after another to severs in round-robing fasion.

Right now only write transactions are used (and can be combined with object creation if needed), read ones are pending as long as multiple parallel transactions (which is not complex, but main task is how to wait them all to be completed, very similar code is used in pohmelfs_aio_read()).

There is also pending task of cache coherency support (server side originated messages to clients, which used the same pages, which another client is writing into, also including metadata coherency messages like uid/gid/inode size and other changes), it is not that complex task, and mostly requires server modifications.

Stay tuned!

/devel/fs :: Link / Comments (0)

Fri, 02 May 2008

Design of the POHMELFS transaction model.

It is heavily based on how netlink is implemented in Linux kernel. Besides the fact that it is likely the most ugly and complex protocol among communication models supported by the kernel, it is exactly the most effective, extendible and feature rich one.
This model is based on the attributes, which are embedded into the message. Each attribute has header, which includes size of the attached data. So, one can put effectively unlimited amount of data into any message (limited only by size field and practical assumptions of the communication), and it is possible to create message, which will contain any number of different attributes.
The main problem of the netlink is its padding and alignment ugliness. Protocol tries to get the every bit out of the communication, so there is huge amount of very hairy things there.

I like to drink and (un)fortunately I got pretty bad quality drinks some times, but I'm absolutely sure, when Alexey Kuznetsov designed netlink attrubute alignment policies he had really bad hangover after likely the ever worst crap he drunk.

So, netlink attributes are very ugly, but you can extend it how you like.
The same applies to POHMELFS transactions.

You can put any new attribute into the transaction in a very trivial manner (I worked with netlink alot, even created kernel connector to simplify kernel development side, so I know that taste), although transaction size is limited, it is controlled only by mount option (default is 32 IO vectors each one of PAGE_SIZE (4k on x86) in one transaction).

Thus one can easily implement for example any protocol security labeling, just add new per-packet attribute.

So, it is easily possible to infinitely extend communication protocol with full backward compatibility.

/devel/fs :: Link / Comments (0)

Tue, 29 Apr 2008

POHMELFS transactions and ACID.

POHMELFS just got initial transactions support and ability to connect to multiple master servers. Master servers are those, which will say, where data is placed. Essentially they are the same severs which may provide that data, but main server addresses are provided during pre-mount configuration time, and data server addresses will be provided by main servers (if main ones will not want to return data) in run-time.
Also main servers can be used to request data in parallel or to switch between them, when curently active one has failed.

So far it is a theory, practice is rather miserable: POHMELFS client connects to multiple servers, but works with only one. Errors are detected, and switch to the next server can happen, but it is not done. Since there is a serious problem with this approach: neither server nor client support ACID for data being written.

Here we come to transaction introduction: it is multiple commands wrapped into single atomic operation. In case of error during transaction write, the whole one will be resent to different server (or the same one after reconnect). This is rather simple (although transactions are not supported by server and client does not wrap any command into it yet), but it still does not solve ACID problem.

Since POHMELFS has writeback cache, all its writes never reach server, instead writeback is scheduled by the system, and it starts writing pages to the server. Current POHMELFS implementation uses only ->writepage() method, which is invoked for each page. It does not require server to return explicit acknowledge, that page was written, instead it relies to underlying transport protocol (like TCP) to handle guaranteed delivery, so data can be queued somewhere when connection was dropped, so POHMELFS client does not know if data was really written or not. Having per-page acknowledge can fix ACID problem realy trivially, but that may (or may not) end up with severe performance degradataion. As a better solution I consider own ->writepages() implementation, where each transaction will contain multiple pages to be written and thus smaller amount of explicit acks from server to be received, and thus smaller performance degradataion. In case of failure whole transaction has to be resent to different server of course.

Server does not support data mirroring to multiple root directories yet, so actually not too much is implemented from above description, but transactions and multiple server connections exist and soon client will get support for reconnection and proper transaction processing.

/devel/fs :: Link / Comments (0)

Sun, 27 Apr 2008

Detailed POHMELFS roadmap.

Transaction support will be added into kernel client. It is possible that it will be exported to userspace (thus it will be synchronous write-through operations).
Also kernel client will get locking support (fcntl() ones first, then more fine-grained ones), this is different from byte-range read/write locking, which will be done on server. It is possible to export it to client too (and will be part of POHMELFS locking API actually, which will be used for fcntl() too).
The simplest case is data invalidation in client's cache (i.e. if one client issued a writeback for given page, it has to be marked as not up-to-date on other clients). Likely it will be done at the beginning of the next week. So far it will be the last cache coherency item. Task is relly simple because of asynchronous processing of all data in kernel client. Server will have to store not only index of directories to watch for object changes there, but also per-object set of pages, read by client, so that appropriate users could be notified, that page is no longer up-to-date and has to be refreshed.

Userspace server will get parallel and distributed facilities. Parallel processing will be done first by allowing lookup and readdir callbacks return inormation about objects, which will contain address of the server where object is actually located, so that server could read, write or check status there. So far the whole file will be stored on a server, i.e. for the first implementation there will not be a possibility to store half of the file on one server and another half on different one. Then it can be extended.
Server will get ability to store data on different root directories (so that client was not able to see shadow copies). There will be simple regexp policies for data storing, for example '*.jpg' has to be stored in root1 and root2, '*.txt' only in root1 and so on. Each root directory can be local or remote mounted one, userspace does not care about this issues.

Main part is already completed: I have a vision of what system has to provide and how it will look like, so with good design of the low-level mechanisms it becomes a doable task for the predictible timeframe.

Stay tuned!

/devel/fs :: Link / Comments (0)

Fri, 25 Apr 2008

POHMELFS release.

Vodka and beer together are glad to provide a new POHMELFS release for you.

POHMELFS stands for Parallel Optimized Host Message Exchange Layered File System.

This is a high performance network filesystem with local coherent cache of data and metadata.
Its main goal is distributed parallel processing of data. Network filesystem is a client transport.
POHMELFS protocol was proven to be superior to NFS in lots (if not all, then it is in a roadmap) operations.

Basic POHMELFS features:

  • Local coherent (notes 1 and 2) cache for data and metadata.
  • Completely async processing of all events (hard and symlinks are the only exceptions) including object creation and data reading.
  • Flexible object architecture optimized for network processing. Ability to create long pathes to object and remove arbitrary huge directoris in single network command.
  • High performance is one of the main design goals.
  • Very fast and scalable multithreaded userspace server. Being in userspace it works with any underlying filesystem and still is much faster than async ni-kernel NFS one.
Roadmap includes:
  • Server extension to allow storing data on multiple devices (like creating mirroring), first by saving data in several local directories (think about server, which mounted remote dirs over POHMELFS or NFS, and local dirs).
  • Client/server extension to report lookup and readdir requests not only for local destination, but also to different addresses, so that reading/writing could be done from different nodes in parallel.
  • Strong authentification and possible data encryption in network channel.
  • Extend client to be able to switch between different servers (if one goes down, client automatically reconnects to second and so on).
  • Async writing of the data from receiving kernel thread into userspace pages via copy_to_user() (check development tracking blog for results).
One can grab sources from archive or check a homepage.

Enjoy!

P.S. Moved to listen blues and drink a beer.

/devel/fs :: Link / Comments (0)

Thu, 24 Apr 2008

Second POHMELFS release.

Is scheduled for tomorrow, today I have to prepare myself for it. The whole idea and implementation started during fun new year vacations, so I have to repeat process at least at some degree...

This release will not include direct writing to userspace from async thread, since this approach happend to be really non-trivial. What I described for the page fault handling works only for the first fault, when page is populated into the table, it can be referenced and written into and thigs just work. Problem happens when the same page used for the second read (i.e. new try from the userspace, for example if to increase size of written data to more than two pages, 'cat' will use the same two pages to read data). With the second write from the kernel there will be page fault again, although page exists in table, and fault can not be handled (at least its reason will not be removed, since it will happen again and again), since page table entry looks really good for the system, but not for the CPU.
I checked two cases: usual copy_to_user() from kernel on behalf of userspace thread invoked a read syscall, and the same code, but copy was performed from the different thread. Page table entry (pte) looks very similar in both cases (in regards of all flags at least), but fault happens for the second write into the same page always, when thread's mm context was changed to point to original userspace one.
This does not change if userspace thread was or was not scheduled away from its CPU.
Difference from get-user_pages() in this part is mainly the fact, that resulted page is locked in the kernel (by increasing its reference counter at least), but I still want to produce the same behaviour as usual page fault during copy on behalf of userspace thread.
So, I stuck with this problem, but since it is very interesting I will find a solution.

Meanwhile, this release will include following things:

  • POHMELFS client. Full client side caching. Async operations for all major events (not including copy_to_user() hack described previously, but just async notifications an copy on behalf of original userspace thread). Support for usual files and directories only, special files like device files or pipes are not interesting at this point, and are quite simple to implement, but so far there is no need for that. Client has support for object creation/removal cache coherency messages.
  • POHMELFS userspace server. Onject creation/removal cache coherency messsage broadcasting will be commented out, no locking.
Stay tuned!

/devel/fs :: Link / Comments (0)

Tue, 22 Apr 2008

Cache coherency in POHMELFS. Continue.

While moving home I thought a lot about cache coherency issues. While we belive that NFS has coherent cache, since it is somewhat write-through, its cache actually is not synchronous, since between object creation and moment when other clients see new object really lot of time can run, for example when client, which create an object, has slow link... So, object creation and removal should not be synced to other clients during writeback on one of them, instead clients which are interested in object perform a lookup, which may or may not return object, this is not a race or cache non-coherency, this is usual multithreaded environment without client's synchronization.

What we really care about, is data consistency on the server. When we have multipage write, which overlaps with another write from different client, we should not read data back from the middle of the transactions. Locking the whole file is not an issue, instead proper byte-range (page-range actually) locking has to be implemented. I already have a prototype, but have to check it in real life.

So, other competing projects may or may not follow my way and drop creation/removal/stat coherency from the TODO list (afacs, no one implemented that yet :) based on my analysis and concentrate on server read/write locking.

And I will start some bits of VM hacking: plan is to implement generic enough (well, working on x86 for start :) mechanism to copy data from different (i.e. not that one which started a syscall) thread to userspace, while original one sleeps in syscall, via copy_to_user(). Likely it will be somewhat similar to what I did for zero-copy userspace sniffer and how get_user_pages() work.
Result, which has to be as fast as usual copy_to_user(), otherwise it is not interesting solution, will be used in POHMELFS client and its async reading.

/devel/fs :: Link / Comments (6)

Mon, 21 Apr 2008

Cache coherency in POHMELFS.

Example:

Client 1			Client 2
# ls -a /mnt/
. ..
				ls -a /mnt
				. ..
				echo qwe > /mnt/asdasd
				sync
ls -a /mnt/
. .. asdasd
rm -f /mnt/asdasd
sync
ls -a /mnt/
. ..
				dmesg | tail -n1
				pohmelfs_remove_response: parent: 2, path: '//asdasd'.
				ls -a /mnt
				. .. asdasd
As you might noticed, when one client creates an object and it is written back to server (during writeback), it is broadcasted to all clients, which read the same directory before. This information is stored on server in binary tree, so it takes (M-1)*O(log(N)) time, where M is total number of clients and N is number of directories they read. This can be further optimized though.

Objects are not removed from clients, when one of them remove it (and this is synced to server via writeback), since so far I can not call sys_unlink() directly from module, and I did not yet wrote code to deal with dentry cache (that will be siple), instead you can see in dmesg, that another clients received a command and just need to drop inode and dentry.

Also inode information is not broadcasted yet (for example when file size increases or access rights are changed), so new files have always zero size. This informaion should be broadcasted during writing, and since server is heavily multithreaded, this should not hurt performance.

There is different opinion though: we do not need cache coherency at all, since the last writer will overwrite data anyway, and when we open new object, we first look it up on server, so if it was created there, it will be opened, but if it exists only in cache on some other client, we do not know about it anyway. We can broadcast above messages during object creation on clients, but this will be effectively write-through cache, since we can create object on server that time.

Anyway, I will proceed with either remove/stat messages, or with ability to copy data to userspace from different thread. The latter looks like very interesting hack.

/devel/fs :: Link / Comments (2)

Sun, 20 Apr 2008

Real Jedi does not use kernel.

He writes new or extends existing, but it is from different serie.

This one will tell you how one will be able to build a distributed and then parallel filesystem using POHMELFS.

Headline says it all: POHMELFS server will not be placed into kernel so far, since it is already very fast (compared to in-kernel async NFS server), and userspace programming is a bit easier and mostly because there is no need to wait about 10 minutes while servers come up after ipmi reboot, since they are located somewhere I do not know where and there is no posibility to quickly reboot them by hand, so servers have lots of things to bring themself up even if something was really screwed, like network boot, add here scsi probing, possible fsck, initial bios memtest (8GB)...

So, planned POHMELFS server updates:

  • PMCC - poor man cache coherency protocol. Scheduled for the first half of the next week, btw.
  • server extension to allow storing data on multiple devices (like creating mirroring), first by saving data in several local directories (think about server, which mounted remote dirs over POHMELFS or NFS, and local dirs).
  • client/server extension to report lookup and readdir requests not only for local destination, but also to different addresses, so that reading/writing could be done from different nodes in parallel.
Somewhere at the beginning there is also a task to extend client to be able to switch between different servers (if one goes down, client automatically reconnects to second and so on).

And the most complex task is server parallelization, i.e. ability to have multiple servers, which handle the same metadata, to work in parallel and be coherent. AFAIK, there are no such (at least open) solutions, neither Lustre, nor PVFS2, nor Ceph, nor glusterfs, nor whatever. There are solutions to have master-slave setup (IIRC, Lustre works that way), Ceph has ability to spread metadata between multiple servers, but they do not handle the same sets of objects, so there is no metadata server redundancy.
So far I consider this as the most complex part, and I have not yet come to solution.

/devel/fs :: Link / Comments (0)

Fri, 18 Apr 2008

Poor man's cache coherency protocol design for POHMELFS.

As you might know, POHMELFS is a network filesystem with client's cache of data and metadata. Any place with cache has to provide cache-coherency algorithm to sync data with other users.

There are two common cases when caches become non-coherent:

  • client created/removed/modified object, which is not shared with other clients (i.e. this object does not exist in theirs caches and no object with the same name was created on different clients)
  • object being handled by one client exists in other caches
Poor man's solution for the above problems resolves quite easily: client will flush its changes to whatever objects it wants during local writeback, this changes are then propagated to all other clients, which worked with parent object (this information will be stored in server each time client read dir or perform a lookup). For the first non-coherent case above client will just receive a new object from the server, which will be easily imported into existing tree (because of async nature of the POHMELFS it is trivial task, which right now works out of the box, although only on client). For the latter case there might be problem if local object was modified: in this case we can either replace its context with new data, or (better) to rename local object to something different (like old name plus sync time), so that user could merge data manually.

So far there will be no locks, which will be implemented next.

/devel/fs :: Link / Comments (0)

POHMELFS AIO reading benchmark vs async NFS.

After I spent two days implemententing real AIO for POHMELFS, following things happened:

  • Implemented 3 different AIO schemes, two of which could be zero-copy. Here is a brief description of them.
    First, POHMELFS ->aio_read() callback schedules number of pages to be read from the server (if page is already up-to-date, it is copied to userspace, otherwise network request is being sent), then it waits...
    • when async data is received from remote side, appropriate inode and pages are found, then (physical) userspace page is locked in memory and data is either received into that page, or received into VFS cache page and then copied into userspace one. Then userspace page is unlocked.
    • when async data is received (note that it is received completely asynchronous in different thread) into VFS cache page, received thread copies data into userspace via copy_to_user(). Since receiver thread has completely different virtual memory layout, it can not simply copy data to provided userspace address, first it has to setup page tables to be equal to userspace thread layout, in theory setting CR3 register on x86 should be enough, but that's only theory, I was not able to fully complete this method, since eventually thread crashed (obviously: userspace thread could be still active on different CPU, so installing the same CR3 register for different CPUs pointing to the same page tables lead to crappy things). This interesting hack can be finished though.
    • when async data is received, pages are marked as ready and placed into list, so userspace thread can copy them back via copy_to_user(). The simplest method. And it works great (graphs below).
  • found a bug in 2.6.25-rc7 shmem when removing 1gb file from it:
    Bad page state in process 'rm'
page:c49948c0 flags:0xf7d4a600 mapping:00000000 mapcount:0 count:0
Trying to fix it up, but a reboot is needed
Backtrace:
Pid: 9454, comm: rm Not tainted 2.6.25-rc7 #11
[] bad_page+0x52/0x7a
[] free_hot_cold_page+0x5e/0x15a
[] __pagevec_free+0x18/0x22
[] release_pages+0xfb/0x142
[] __pagevec_release+0x15/0x1d
[] truncate_inode_pages_range+0xea/0x29f
[] __link_path_walk+0xa7e/0xb28
[] truncate_inode_pages+0x9/0xc
[] shmem_delete_inode+0x26/0xac
[] shmem_delete_inode+0x0/0xac
[] generic_delete_inode+0x88/0xec
[] iput+0x60/0x62
[] do_unlinkat+0xb7/0xf9
[] do_page_fault+0x2b6/0x6c2
[] do_page_fault+0x31e/0x6c2
[] sys_ioctl+0x2c/0x43
[] sysenter_past_esp+0x5f/0x85
[] pci_scan_single_device+0x377/0x446
    Did not try to investigate (this is my testing server, not tainted with POHMELFS code).
  • Ran multiple tests...
Test details for the second round of POHMELFS vs NFS fight.
Hardware and software was already described in the first round, I need to note, that server (2.6.25-rc7) has all debugging options turned off.

Tests performed: kernel tree reading (find linux-2.6.24.4 -type f | xargs cat > /dev/null) from disk over the net (XFS filesystem, cold server and client caches) and big file reading from the tmpfs (to eliminate server disk latencies). Graph was added to the previous round results.

POHMELFS vs NFS

Note that async NFS and POHMELFS behave very similar with operations which involve reading from the disk, that is because of disk latencies (although 10krpm SCSI disk used allows about 80 MB/s sequential read, XFS behaves quite badly with lots of small files), tmpfs comparison shows advantages of the POHMELFS network protocol.

Reading from huge remote tmpfs file is about 2 times faster for POHMELFS because of its AIO implementation, although it is not main reason - server was almost always capable of handling requests from the POHMELFS client one-by-one using one thread, which saturated bandwidth for about 70% (add here all debug options turned on on client). One of the main factors I think is readahead being turned off - sync readahead has zero advantage in asynchronous network filesystem, since while it waits for readahead to complete, it could schedule new requests, while ->readpage() method used in readahead waits until page is transferred, and only then readahead code schedules new request. One can implement ->readpages() though.

Kernel tree reading micro-benchmark was also performed: POHMELFS has 2-times win because of its network protocol, which batches (via TCP_CORK only though, I think I need to implement better directory reading command) server replies.

Another solution is to correctly implement transactional model, which is next task now.

/devel/fs :: Link / Comments (0)

Wed, 16 Apr 2008

Massively multithreaded POHMELFS server.

Because of completely asynchronous POHMELFS nature it is possible to implement mulithreaded server, where not only requests from different clients are processed in parallel, but also async requests from the same users are handled simultaneously by pool of threads.
Such multithreading requires to introduce transactional model of the communications, for example object creation and writing data, right now this race is handled by sending a reply after creation, so the whole writeback sleeps waiting for that, which drops performance (to NFS level). Transaction contrary will contain both operations, which will be processed by the same thread without race. It can also handle other problematic places with multiple server threads.

So far userspace server can run several or one processing thread per client, but there is no transactions implemented. I just started AIO reading implementation, which should provide great speedup for any reading workload.

Stay tuned!

/devel/fs :: Link / Comments (0)

Mon, 14 Apr 2008

Initial network filesystem benchmark. POHMELFS vs NFS. Round 1.

Hardware (both client and server have the same hardware).
4-way (2 logical (HT) + 2 physical cpus) 3.00 Xeon (32 bits with PAE :), 8 GB of RAM, Intel 82541GI gbit adapters, Seagate ST3300007LC 10k rpm scsi disk on Adaptec AIC7902 PCI-X Ultra320 SCSI adapter.

Software.
Server: 2.6.25-rc7 kernel, in-kernel NFS server, userspace POHMELFS server.
Client: 2.6.25-rc8 kernel, in-kernel clients.
Both have all kernel debugging turned on.

Round 1. Huge directory (linux-2.6.24.4.tar archive) untarring over the network.
Picture shows it all.

POHMELFS vs NFS

Notice, that there is no test for POHMELFS reading (that is why it is only first round), since it is miserable. And I know the reason: I'm lazy, so I use generic reading function (generic_file_aio_read()), but actually Linux does not have AIO reading from usual files, so it is very synchronous and requires to read data page-by-page, so we have a pretty broken system in regards to network performance.

Since reading is not async, so I will reimplement generic_file_aio_read() as pohmelfs_aio_read(), which will be a real AIO reading function. That will be second round, where POHMELFS will win.

But it can not win the game. Because things are changing. Today I've known, that if filesystem has only 20 users over the world, then it should not be merged, since burden of changing something generic in VFS (and thus propagate it to filesystems) is too high.

What has happend? Linux kernel maintainers started to be afraid of changes? Afraid of more code? Afraid of something new they do not want?..

Eh, and they tell they want more developers... They want monkeys who will do only what was asked them to do.

POHMELFS will be sent for review of course, but it is highly unlikely I will push it upstream.

/devel/fs :: Link / Comments (6)

Fri, 11 Apr 2008

Unhashed inodes can not be synced during writeback. Debunked.

Problem happend to be quite simple: writeback happens for inodes in sb->s_io superblock list. They are placed there from sb->s_dirty list, which contains dirty inodes. Dirty inodes can be placed into that list via mark_inode_dirty(), which checks if inode is hashed, if it is not, then it will not be placed into dirty list.
Hashed has a synonym in comments: valid...

There is sb->s_op->dirty_inode() superblock operation callback, which is invoked first, so one can still implement own inode cache, do not use inode hash tables, do not hash inodes and still put inodes into dirty list and thus be able to run writeback on them.

/devel/fs :: Link / Comments (0)

Thu, 10 Apr 2008

Busy inodes after unmount. 

VFS: Busy inodes after unmount of pohmel. Self-destruct in 5 seconds.  Have a nice day...
After removing private cache of inodes I found, that objects, which were sent by the server and which were never attached to directory entry (dentry), will never be freed.
So, essentially this does not work with Linux VFS:
iget()/iget_locked()
...
umount
Inodes, created by iget()/iget_locked() will be placed into at least three different lists:
  • inode_in_use - global list of ever created inodes, which have i_count and i_nlink more than 0
  • s_inodes - per superblock list, which contains every inode, created for this superblock
  • inode_hashtable - hash table indexed by inode number. If you want to work with writeback, your inodes have to be there. Did not yet investigate why.
So, essentially all inodes, which you created, are accessible by VFS and will be checked during umount via generic_shutdown_super()->invalidate_inodes(), where system will notice that if inode in s_inodes list has non-zero reference counter (or course, otherwise it would be already freed by filesystem), then this inode can not be freed. Thus we have a leak.

Above lists can only be accessed under global inode lock, so it is not a good idea to destroy inodes traversing them in for example ->put_super() callback or in any other filessytem callback, so I had to add a list of all inodes into POHMELFS superblock. Ugly.

/devel/fs :: Link / Comments (0)

POHMELFS development status.

It has developed very rapidly last couple of days, so essentially I rewrote it. I think it is ready for the next release, which I will announce in a day or so.
Right now all first-milestone features except cache-coherency (check below), which I planned, are completed (although maybe not in the most optimal way sometimes).
Because of name cache usage it is now possible to create huge pathes with multiple directories via single command. The same applies to directory removal, although it is because of different design issue.
It would be possible to rewrite generic read/write helpers and provide set of pages into POHMELFS network stack (which is page based for data now), but I decided that for the first step it is not needed.
POHMELSF has now fully async processing of all operations except link creation (I just decided that it is a bit simpler to make them write-through, it was done because of laziness and not some fundamental arch problems). It was achieved by serious (read: from scratch) changes in the arch, which had own problematic places, namely error report. Because of this move it becomes really simple to implement any kind of protocol, if it obeys async rules, namely sending of the message never requires sync reply, and where it is needed, reply comes as an independent incoming message, which is processed asynchronously from waiting and via common state machine.
Such arch allows to have simple cache coherency algorithm, when server just sends a missed entries or commands to remove some objects and client's core handles that just fine since its reciving code does not depend on sending one. This is not 100% correct way to handle collisions (collisions thus became new objects in the filesystem tree, like old name plus some suffix), but it is what lots of the users need, but not real cache-coherency.
Writeback cache does not play very well with cache-coherency, since every metadata changes (like object creation or removal) has to be checked against server state, since different clients can do the same with the same object. Level of paranoidality has to be thought of in advance.

First cache-coherency step is implementation of the trivial scheme, when every object is synced during its writeback time and changes being broadcasted by server to other clients. If another client has the same object being processed it can either be renamed to collision or just overwritten. Having locks and thus real states is a next step.

Also, POHMELFS does not have authentification and strong checksums right now, and although this is a simple task to implement, its priority is questionable. There is also possibility to implement cryptographically strong encryption of the communication channels.

So, lots of ideas, but main part is ready - async data processing design was definitely a right choice to implement, so all other features become very simple to complete.
New release will be announced very soon, stay tuned!

/devel/fs :: Link / Comments (0)

Sun, 06 Apr 2008

The is only one way: asynchronous.

This is a new motto for POHMELFS. It is a completely new filesystem now.

POHMELFS got new page processing code (sending side: commands and data), new lookup, which is based on the Linux VFS inode cache without reinventing the wheel (comment says it is very smp-friendly, although I do not quite understand how it is possible with global inode_lock), it also got completely new object creation and referencing path. It is possible to create a huge path (up to 4k, but can be easily extended if there will be such demand) with multiple objects in it with only single network command.
But the main feature of new POHMELFS is its name cache. I did not find how to hook into VFS dentry cache, so invented own. It is fast to travers from child to the highest level parent, which is actively used in POHMELFS writeback path. Although it is not 100% the best storage, but a simple RB-tree (and thus requires smp-unfriendly mutex), the whole idea shows its gains already. Eventually it will be replaced with faster and more scalable approach protected by RCU (even properly sized hash table will show better scalability, although dynamic resizing of hash tables prevents RCU usage), but I started from the simplest ground.

POHMELFS already outperforms async NFS during untarring and completely saturates my testing Xen domains (both network and disk speed), while NFS is almost two times slower. Testing machines have 256 Mb of RAM, maximum 3 MB/s interconnect speed (something is broken in Xen setup likely, since it is supposed to be 100 mbit/s and there is no high load), which is very unfriendly (read: in such scenario POHMELFS will show its worse results) for POHMELFS, but nevertheless it is fast.

It became not only much faster, but also simpler. Its userspace server has two times less lines of code (816 vs. 1613), kernel side is smaller and simpler too: mainly there are no zillions of different trees indexed by any possible keys, so far only per-inode tree of child names for readdir and per-superblock path entry cache.

There are drawbacks of course: there is no receiving code (at all). It will be a dedicated thread, which will asynchronously process all incoming packets (mostly readdir async return, read page content and cache-coherency messages). First two are really simple. The last one will be implemented as a full MOSI/MSI library for inode content. Likely it will be possible to use in my other projects.

P.S. I frequently think that I'm very good vapourware seller :)
Stay tuned!

/devel/fs :: Link / Comments (0)

Wed, 02 Apr 2008

Unhashed inodes can not be synced during writeback.

So essentially there is no way to implement own inode cache tied to system's writeback mechanism, which is a bad news. POHMELFS in its current reincarnation does not use system's inode cache and all its indeas are unhashed, which results in a fact, that they are never synced, since writeback mechanism just does not see them.
So I will fallback to hashed inodes, which will be used just for that, and writeback for single inode will end up creating directory structure for the all upper layer objects.

Another idea is to implement own writeback, which would be scheduled from the main one or after memory notifications, this approach has lots of advantages actually, but let's first complete simpler part with hased inodes.

This is called learning curve - I'm essentially where I was before, but with extended baggage of knowledge.

/devel/fs :: Link / Comments (0)

Sun, 30 Mar 2008

To SSD or not to SSD.

Couple of days ago I talked with person, who ordered 4 high-end 128G SSD disks to create RAID for testing purposes, seek time for that devises is 0.1ms. Each one costs about $4k. His main workload is databases, i.e. random reads and writes, so we calculated that theoretically it has to be about 14 times faster than high-end scsi disks with 3.5 ms seek latency and about 100Mb/ssequential access speed in given workload for processing random data at 8-16kb chunks (usual 'page' in sql servers). Besides the fact, that putting 14 disks into mirror will be as fast as single ssd disk (theoretically), it will be 14 times more reliable and likely have smaller price, main workload is to replace RAM with SSD, not disks with SSD.

My prognosis is that SSD will be at most 2-3 times faster (if will be fater at all, since its theoretical performance advantages can be killed by FS) than SCSI disk for given workload, and as is, it is not a breakthrough technology.

If I'm wrong (it will be tested likely next week with sysbench read-write benchmark), I will buy a good bottle of whiskey for us, otherwise...

/devel/fs :: Link / Comments (5)

Thu, 27 Mar 2008

Filesystem as a database or database in filesystem.

I actually do not understand what prevents filesystem writers to implement trivial interface and access library for metadata manipulations, which would allow not only path lookup, but also lookup for various keys, for example stored in extended attributes. Yes, it requires filesystem changes, but I can not believe it is impossible or even too complex.
Need to think...

/devel/fs :: Link / Comments (2)

Wed, 26 Mar 2008

Added maildir benchmark results.

The simulation works on each filesystem in the following stages:

  1. The empty filesystem is created and mounted.
  2. The directory structure is created, with no files.
  3. A single delivery simulator and retrieval simulator are run simultaneously. The script waits for each of the simulators to finish, and then runs the sync command before proceding to the next step.
  4. The above step is repeated with 2, 4, 8, and then 16 delivery simulators.
Delivery Simulator.
The delivery simulator does actual maildir deliveries to the given directory:
  1. It writes a file with a unique file name to the tmp subdirectory.
  2. It fsyncs the newly written file.
  3. It renames the file into the new subdirectory.
  4. It fsyncs the new subdirectory (to ensure that directory is actually on disk, as most Linux filesystems don't automatically perform this action during the rename).
More details on original page.

Briefly saing, it is multithreaded maildir simulation.
And results are quite different compared to for example postmark: very good results from xfs, jfs and reiserfs.
There are no ext2 and btrfs filesystems, since perl's fsync says that filedescriptor opened there is invalid: Invalid argument at /root/fs_bench/maildir_fsbench/fsbench/fake-deliver line 38.
Interested reader can check sources and show me a problem, but ext2 worked pretty fine with 2.6.20 kernel and to date glibs/perl/whatever was in Debian.

Anyway, results can be found at contest homepage.

Now all testing is over.
Main conclusion: things got worse compared to 2.6.20 and there was no major breakthrough in filesystem development at least from perfomance point of view.

/devel/fs :: Link / Comments (0)

Additional XFS test with slightly dif